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RECOGNITION-PRIMED  DECISION  STRATEGIES 


EXECUTIVE  SUMMARY 


The  objective  of  this  three-year  research  contract  was  to  increase  our 
understanding  of  naturalistic  decision  making  (NDM).  Major  goals  include  an 
examination  of  the  Recognition-Primed  Decision  (RPD)  m^el  for  implications  about 
processes  of  situation  assessment  and  mental  simulation,  exploration  of  limitations  of 
the  RPD  strategy,  and  identification  of  methods  for  supporting  decision  makers. 
Several  studies  addressed  situation  assessment  in  decision  making,  and  we  learned 
about  the  importance  of  mental  simulation  for  formulating  and  evaluating  situation 
assessment;  we  also  identified  a  set  of  factors  that  appear  critical  for  formulating  and 
communicating  situation  assessment.  In  reviewing  the  limitations  of  a  recognitional 
decision  strategy,  we  identified  an  inherent  problem  whereby  people  can  use  mental 
simulation  to  explain  away  discrepant  data,  thereby  preserving  an  inaccurate  situation 
assessment.  In  developing  strategies  for  supporting  recognitional  decision  making,  we 
tested  a  technique  for  using  mental  simulation  to  identify  weaknesses  in  a  planned 
course  of  action,  thereby  reducing  over-confidence. 

In  accomplishing  these  objectives,  there  have  been  a  number  of  important 
outcomes  that  were  not  tied  to  specific  initial  goals.  First,  we  feel  that  our  work  has 
helped  to  establish  naturalistic  decision  making  as  an  important  perspective.  The 
previous  traditions  of  classical  decision  research,  drawing  on  mathematics,  statistics, 
and  game  theory,  have  definite  strengths  and  weaknesses;  but  there  is  now  emerging 
an  acknowledgement  of  the  importance  of  NDM  for  addressing  the  problems  and 
requirements  that  are  not  handled  well  by  the  classical  research  strategies.  The  1989 
workshop  on  NDM,  supported  in  part  by  this  research  contract,  served  to  crystalize 
this  approach,  and  the  forthcoming  book  Decision  Making  in  Action:  Models  and 
Methods  will  continue  that  process.  Without  these  projects,  the  earlier  research 
findings  would  likely  have  remained  on  the  periphery  of  the  decision  research 
community.  These  projects  have  implications  for  the  direction  of  decision  research. 
They  have  led  to  an  expansion  of  cognitive  accounts  of  decision  making  with 
important  applications  for  decision-centered  design  of  interfaces  and  displays,  training 
that  emphasizes  situation  assessment,  mental  simulation  and  metacognition  rather  than 
analytical  decision  strategies,  and  more  effective  approaches  to  team  decision  training. 

Second,  we  have  demonstrated  that  the  RPD  model  generates  empirical  as  well 
as  ethnographic  research.  While  the  RPD  model  was  primarily  developed  using 
ethnographic  research  methods,  there  have  been  as  many  empirical  studies  of  the 


V 


model  as  ethnographic  studies.  The  field  studies  have  taken  place  in  diverse  settings 
(urban  firefighting,  forest  fire  management,  design  engineering,  tank  platoon 
operations,  nursing  in  Intensive  Care  Units),  and  the  incident  accounts  from  these 
studies  have  been  vivid  and  memorable,  so  it  may  be  easy  to  forget  the  empirical 
studies.  Thus  far  there  have  been  at  least  a  half-dozen  empirical  studies  of  the  RPD 
model.  We  have  examined  firefighters,  using  multi-dimensional  scaling  (Calderwood, 
1989);  we  have  used  prepared  scenarios  to  collect  decision  protocols  from  experienced 
and  inexperienced  fireground  commanders  (Calderwood,  Crandall,  &  Klein,  1987). 
Mosier  (1990)  has  confirmed  the  use  of  recognitional  decision  making  by  airline 
pilots,  using  analyses  of  videotapes  collected  during  high  fidelity  simulations.  We 
have  shown  that  proficient  performance  can  be  maintained  by  skilled  chess  players, 
despite  severe  time  constraints  (Calderwood,  Klein,  &  Crandall,  1988).  We  have 
shown  that  skilled  chess  players  generate  superior  options  as  the  first  ones  they 
consider,  a  direct  test  of  the  RPD  model  (Klein,  Wolf,  Militello,  &  Zsambok,  in 
preparation).  And  we  have  shown  that  mental  simulation  exercises  can  improve 
decision  processes  of  college  students  engaged  in  planning  tasks  (Kyne,  Militello,  & 
Klein,  in  preparation). 

Third,  through  the  mix  of  ethnographic  and  empirical  research,  the  RPD 
model  has  become  more  mature  and  comprehensive  than  it  was  three  years  ago.  It 
has  become  elaborated  to  portray  several  means  of  achieving  situation  assessment.  It 
has  been  shown  to  generate  testable  hypotheses,  and  it  has  been  experimentally 
supported. 

Fourth,  the  primary  expansion  of  the  RPD  model  has  been  to  achieve  a  greater 
understanding  of  the  process  of  mental  simulation.  This  current  research  effort  has 
accomplished  that  important  objective.  We  have  derived  a  model  of  mental 
simulation  that  is  far  more  detailed  than  any  previously  available,  and  we  have  shown 
how  mental  simulation  serves  as  a  source  of  power  for  a  variety  of  cognitive 
functions. 
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INTRODUCTION 


In  1985,  Klein  Associates  performed  a  study  for  the  Army  Research  Institute  to  find 
out  how  people  can  make  satisfactory  decisions  under  time  pressure,  when  diey  don’t  have 
enough  time  to  generate  and  evaluate  a  large  set  of  options.  The  investigation  centered  on 
decision  making  of  fireground  commanders.  We  found  that  the  commanders  rarely  had  to 
generate  more  than  one  option,  even  for  difficult  cases.  They  were  using  an  approach  that 
we  caUed  a  Recognition-Primed  Decision  (RPD)  strategy  for  making  rapid  decisions.  The 
RPD  strategy  reflects  naturalistic  constraints  on  decision  making  that  are  not  addressed  by 
classical  decision  strategies. 

During  the  past  three  years  we  have  conducted  a  Basic  Research  project  aimed  at 
understanding  the  role  of  experience  and  competence  for  naturalistic  decision  making 
(NDM).  We  have  specifically  examined  the  RPD  model  which  describes  a  decision  strategy 
that  appears  well  suited  for  operational  settings  marked  by  time  pressure,  ambiguity , 
incomplete  information,  ill-defined,  and  shifting  goals.  The  revised  research  plan  submitted 
to  our  sponsor  in  March,  1989  listed  three  goals;  Evaluate  the  relative  strengths  and 
weaknesses  of  the  RPD  strategy;  learn  more  about  the  nature  of  situation  assessment  in 
command-and-control  environments;  and  determine  whether  the  RPD  strategy  has 
implications  for  improving  decision  quality. 

The  purpose  of  this  Final  Report  is  to  document  what  we  have  learned  in  conducting 
the  project  and  pursuing  these  goals.  Each  goal  has  been  accomplished,  although  there  is 
much  more  to  be  learned  about  decision  making  in  operational  settings.  We  follow  with  a 
brief  account  of  the  background  that  shaped  our  investigations.  Then  this  report  wiU  trace 
the  development  of  the  RPD  model,  from  earlier  contracts  through  the  present  one.  Once  we 
have  established  this  framework,  we  will  turn  to  more  specific  accounts  of  the  research 
performed  under  this  contract.  The  topics  wiU  cover  mental  simulation,  situation  assessment, 
option  generation,  errors  and  stress,  and  team  decision  making,  as  well  as  applications  of  the 
models  and  findings  of  our  research.  The  common  theme  runmng  through  these  topics  is  the 
attempt  to  learn  how  people  are  able  to  use  experience  to  make  decisions  under  difficult 
conditions. 


Background 


Classical  research  on  decision  making  (see  reviews  by  Beach  &  Lipshitz,  in  press, 
and  by  Cohen,  in  press)  has  demonstrated  the  power  of  a  variety  of  quantitative  methods  for 
decomposing  and  analyzing  complex  decision  tasks.  These  methods  certainly  are  useful  in 
naturalistic  contexts  such  as  trying  to  decide  which  type  of  bullet  should  be  acquired  by  a 
police  department  (Hammond  &  Adelman,  1976).  However,  for  many  decision  tasks,  the 
time  pressure  and  limited  resources  available  make  it  difficult  or  impossible  to  conduct  such 
analyses.  Accordingly,  Beach  and  Mitchell  (1978)  and  Payne  (1976)  presented  a  contingency 
model  of  decision  making,  showing  that  different  types  of  strategies  would  be  used  under 
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different  conditions.  Svenson  (1979)  has  documented  a  range  of  noncontingent  decision 
strategies,  which  enable  a  person  to  select  a  course  of  action  without  going  through  a  full  set 

of  analyses. 

At  the  same  time,  Kahneman  and  Tversky  (1982)  had  been  examining  the 
consequences  that  can  arise  when  people  use  heuristics  to  substitute  for  a  full  analysis  of 
probabilities  and  contingencies.  They  do  not  paint  a  pretty  picture.  The  heuristics  result  in  a 
number  of  biases  that  have  been  well-documented  during  the  last  20  years.  Accordingly,  we 
are  left  with  a  discouraging  picture  of  quantitative  methods  that  are  rarely  used  because  of 
the  information  processing  limits  of  decision  makers,  and  a  set  of  biases  that  seem  inevitable 
when  heuristics  are  used  to  substitute  for  stronger  methods.  Experts  are  just  as  susceptible  to 
the  biases  as  novices. 

Orasanu  and  Connolly  (in  press)  project  a  different  perspective.  They  point  out  that 
just  as  we  have  been  losing  confidence  in  decision  makers  because  of  their  reliance  on 
heuristics,  the  field  of  problem  solving  has  demonstrated  over  and  over  again  the  power  of 
these  heuristics  and  the  clear  differences  between  experts  and  novices.  They  define  a  set  of 
conditions  that  encompass  NDM,  including  time  pressure,  ill-defined  and  shifting  goals,  and 
ambiguous  and  incomplete  information.  These  conditions  usually  make  it  impossible  to  use 
rigorous  quantitative  methods,  yet,  somehow,  we  are  able  to  function  fairly  well.  One  of  the 
key  questions  arising  from  the  NDM  perspective  is  how  people  are  able  to  use  experience  to 
handle  difficult  conditions. 

The  RPD  model  was  derived  from  interviews  with  fireground  commanders,  to  learn 
their  strategies  and  account  for  their  ability  to  use  experience  to  handle  extreme  time  pressure 
and  other  NDM  characteristics.  The  next  section  describes  the  RPD  model  and  its 
development. 


DESCRIPTION  OF  THE  RPD  MODEL 

We  originally  formulated  the  RPD  model  to  explain  how  experienced  decision  makers 
could  function  in  time-pressured  situations  where  they  were  not  able  to  generate  and  compare 
large  sets  of  options.  The  RPD  model  fuses  two  processes,  situation  assessment  and  mental 
simulation,  to  explain  such  decision  making.  The  model  asserts  that  experience  enables 
decision  makers  to  recognize  the  essential  characteristics  of  a  situation,  and  thereby  to 
identify  feasible  goals  and  plausible  courses  of  action.  Courses  of  action  are  generated  and, 
if  necessary,  evaluated  one  at  a  time,  using  mental  simulation  to  see  if  there  are  any  pitfalls 
to  carrying  out  the  action  within  the  context  of  the  situation.  Of  course,  people  sometimes 
have  to  wrestle  with  different  options  to  pick  the  best  one,  a  task  we  refer  to  as  concurrent 
(as  opposed  to  serial)  evaluation;  concurrent  evaluation  of  options  is  the  subject  of  other 
models  of  naturalistic  decision  making,  and  is  outside  the  scope  of  the  RPD  model. 

There  are  a  few  misperceptions  about  the  model  that  should  be  addressed.  First,  the 
model  does  not  claim  that  people  just  retrieve  courses  of  action  from  memory.  In  many 
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cases,  a  decision  maker  wiU  have  to  construct  a  unique  course  of  action.  Recognition 
provides  a  sense  of  reasonable  goals,  and  a  sense  of  the  typicality  of  the  situation.  In  some 
cases,  there  may  be  a  direct  analogue  enabling  the  decision  maker  to  recall  a  specific  course 
of  action  that  worked  before.  In  other  cases,  a  course  of  action  will  be  partially  constructed 
and  partially  based  on  retrieval  of  actions  from  similar  incidents.  The  RPD  model  simply 
claims  that  options  are  not  generated  randomly,  and  that  situation  assessment  lets  the  decision 
maker  identify  promising  actions  as  the  first  ones  considered. 

Second,  the  RPD  model  includes  aspects  of  problem  solving  and  judgment  along  with 
decision  making.  In  the  domains  we  have  studied,  the  three  processes  are  closely  related. 
Furthermore,  it  appears  as  if  the  skill  of  experienced  decision  makers  enables  them  to  bypass 
having  to  perform  a  concurrent  evaluation,  so  they  do  not  have  to  select  one  option  from  a 
set  of  possible  courses  of  action. 

Third,  the  RPD  model  has  evolved  over  the  past  few  years,  but  it  has  not  changed. 
That  is,  the  framework  of  the  model  has  not  been  revised.  Rather,  the  model  has  become 
more  elaborated  and  sophisticated  as  it  has  been  applied  and  tested  in  different  domains. 

RPD:  Chronology 


Our  initial  research  on  time-pressured  decision  making  was  not  driven  by  any 
particular  theoretical  model.  We  attempted  to  face  the  paradox  of  how  people  can  make 
good  decisions  in  limited  time.  Classical  models  such  as  MAUA  assume  lengthy  and 
extensive  option  generation  prior  to  evaluation  and  selection,  but  under  time  pressure,  this 
seems  implausible.  Somehow,  the  option  set  must  be  reduced.  But  how  can  this  be  done? 

If  the  option  set  is  reduced  blindly,  there  is  a  chance  that  the  person  will  have  an  inferior  set 
from  which  to  choose,  and  will  make  a  poor  decision.  If  the  option  set  is  pre-screened,  how 
might  this  happen  intelligently,  except  by  first  generating  all  options  and  eliminating  the  poor 
ones?  But  this  strategy  requires  the  massive  generation  of  options  that  it  was  supposed  to 
avoid.  So  we  saw  no  way  for  a  person  to  successfully  run  a  MAUA  on  a  limited  set  of 
options. 

The  hypothesis  we  used  was  based  on  the  work  of  Soelberg  (1967;  see  Montgomery, 
in  press,  for  a  recent  version),  who  claimed  that  decision  makers  quickly  recognized  a 
favorite  option,  and  a  next-best  option,  and  that  the  decision  making  consisted  of  trying  to 
show  that  the  favorite  option  dominated  the  next-best  option  on  all  evaluation  dimensions. 

This  hypothesis  would  account  for  effective  rapid  decision  making,  and  in  our 
research  with  fireground  commanders  (Klein,  Calderwood,  &  Clinton-Cirocco,  1986)  we 
attempted  to  find  evidence  for  it.  We  failed.  Based  on  32  fire  or  emergency  incidents,  we 
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probed  156  decision  points,  and  we  found  that  firefighters  claimed  they  were  generating  only 
a  single  option,  for  most  cases.  ^ 

In  its  earliest  form  (Klein,  Calderwood,  &  Clinton-Cirocco,  1985,  1986)  the  RPD 
model  was  simply  the  synthesis  of  the  claims  of  the  fireground  commanders  we  interviewed 
Figure  1  shows  the  initial  version  of  the  RPD  model.  Note  that  the  essential  components  of 
the  model  are  in  place:  situation  assessment  to  frame  the  decision  (prototype  match)  and 
generate  a  plausible  option,  and  mental  simulation  (evaluate)  to  implement,  modify,  or  reject 
an  option.  For  cases  where  the  initial  option  is  judged  unsatisfactory,  additional  options  are 
generated  and  tested  in  a  serial  fashion.  This  stands  in  contrast  to  classical  decision  mode  s 
that  emphasize  concurrent  evaluation  of  option  sets.  The  RPD  model  explains  how  people 
can  use  experience  to  make  effective  decisions  without  having  to  generate  and  then  prune 
option  sets.  This,  in  turn,  explains  how  people  can  make  effective  decisions  under  time 
pressure.  It  must  be  remembered  that  this  was  the  goal  of  the  RPD  model--to  explam 
effective  decision  making  that  does  not  require  concurrent  evaluation  of  options. 


Implement 

Prototype  Match - Action  Queue - Evaluation —  Modify 

Goals 

Perceptual  —Expectations 

Causi  Factors 

Figure  1.  Recognition-Primed  Decision  (RPD)  Model:  1985. _ 

Supported  by  a  Basic  Research  Contract  with  ARI,^  we  attempted  to  replicate  the 
findings  of  the  initial  research.  Calderwood,  Crandall,  and  Klein  (1987)  studied  expert  and 
novice  fireground  commanders,  and  concentrated  on  only  nonroutine  decision  pomts  within 
nonroutine  incidents.  The  basic  findings  held  up.  We  also  found  that  we  could  distmguish 
among  three  types  of  strategies.  In  the  simplest  case,  the  recognitional  match  allowed  the 
person  to  generate  an  option  and  did  not  lead  to  additional  effort  at  situation  assessment  or 
mental  simulation.  We  called  this  the  automatic,  or  simple  RPD.  The  next  case,  a  verified 
RPD,  occurred  when  the  person  was  unsure  about  the  nature  of  the  situation  assessment, 
devoted  some  conscious  attention  to  it,  and  also  did  some  mental  simulation  to  evaluate  the 
option  generated.  The  complex  RPD  strategy  occurred  when  the  person  needed  to  revise 
situation  assessment,  or  when  the  person  needed  to  reject  an  option,  based  on  the  mental 
simulation,  and  generate  additional  options.  These  three  variants  are  shown  in  Figure  2.  It 
was  in  this  study  that  we  began  to  appreciate  the  importance  of  pre-decision  problem  solving 
and  framing  issues  affecting  situation  assessment.  Situation  assessment  was  portrayed  as 
involving  goals,  cues,  expectancies  and  options;  expectancies  were  shown  to  lead  to 


^This  research  was  supported  by  an  SBIR  Phase  I  Contract  No.  MDA903-85-C-0099 
with  U.S.  Army  Research  Institute  (ARI). 

^Basic  Research  Contract  No.  MDA903-85-0327. 
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re-examination  of  the  situation  in  the  case  of  surprises.  Also,  we  described  the  function  of 
mental  simulation  in  greater  detail,  to  show  the  possibUities  of  modification  or  rejection  of  an 

option. 

This  was  where  the  RPD  model  stood  in  1988,  at  the  end  of  our  Basic  Research 
contract. 


(a)  Automatic  RPD  (b)  Verify  RPD 


Cues  &  Knowledge 


Recognize 

Typicality 


CT  Goals  ^ 
(^[^tical  Cues^ 


(^[Expectanci^]^ 


Typical 

Reaction. 


Implement 
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(c)  Serial  RPD 
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Action 
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Figure  2.  Recognition-Primed  Decision  model:  1988. 
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RPD:  Boundary  Conditions 


The  RPD  model  does  not  cover  all  of  naturalistic  decision  making.  What  is  not 
accounted  for  by  the  RPD  model? 

•  Cases  where  the  person  lacks  an  experience  base  to  recognize  situation 
assessments,  and  must  rely  on  decompositional  methods.  Rasmussen  (1985)  has  referred  to 
these  as  knowledge-based  behaviors. 

•  Cases  where  several  options  are  generated  and  the  person  uses  analytical  methods 
to  select  the  best.  Examples  here  are  Elintination-by-Aspects  (EBA)  (in  which  the  evaluation 
process  is  decomposed  into  single  categories,  applied  one  at  a  time  in  descending  order  of 
importance);  Multi- Attribute  Utility  Analysis  (l^UA)  (the  same  as  EBA  except  all 
categories  are  applied  at  the  same  time,  to  gain  the  benefits  of  compensational  trade-offs). 

•  The  Search-for-Dominance  Structure  strategy.  This  strategy  is  described  in  one  of 
the  chapters  in  the  book  Decision  Making  in  Action:  Models  and  Methods,  (Ablex 
Publishing  Corporation,  in  press).  SDS  involves  concurrent  evaluation  of  options. 

RPD:  Extensions  and  Refinements 

The  RPD  model  has  been  elaborated  during  the  last  few  years,  primarily  through 
projects  supported  by  the  Basic  Research  contract  covered  by  this  report.  There  have  been 
major  theoretical  elaborations  of  the  RPD  model  in  two  areas:  mental  simulation  and 
situation  assessment.  These  are  described  next. 

Mental  Simulation^ 


How  is  mental  simulation  different  from  other  forms  of  reasoning,  or  from  rule-based 
accounts  of  behavior?  Clearly,  the  phenomenon  overlaps  other  forms  of  inference  and  yet  it 
appears  to  have  its  own  underlying  characteristics.  For  this  reason,  it  is  important  to  provide 
a  more  detailed  account  of  mental  simulation  so  that  its  unique  characteristics  can  be  better 
appreciated,  its  role  in  decision  making  better  understood,  and  its  heuristic  power  trained  and 
applied. 

In  our  view,  an  important  outcome  of  this  research  effort  has  been  the  gains  in 
understanding  of  the  phenomenon  of  mental  simulation.  We  define  mental  simulation  as  the 
process  of  mentally  enacting  a  sequence  of  events.  People  frequently  construct  and  carry  out 
mental  simulations  in  the  course  of  problem  solving  and  decision  ma^g.  We  believe  that 
mental  simulation  is  an  important  source  of  power  in  operational  settings  characterized  by 
ambiguous  or  incomplete  information,  time  constraints,  and  high  risk.  Within  the  RPD 


^The  term  "mental  simulation"  was  introduced  into  the  decision  literature  by  Einhom  and 
Hogarth  (1981).  (See  Klein,  1990a;  also  Klein  &  Crandall,  in  preparation,  for  a  more 
detailed  examination). 
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model,  mental  simulation  serves  a  variety  of  essential  functions,  including  evaluating 
situation  assessment,  learning  about  the  dynamics  of  the  situation,  evaluating  courses  of 
action,  improving  courses  of  action,  and  generating  expectancies. 

We  believe  that  there  are  additional  functions  of  mental  simulation  for  planning  and 
decision  making.  This  aspect  of  recognitional  decision  making  may  be  especially  vulnerable 
to  time  pressure  and  other  stressors.  For  example,  a  critical  function  of  mental  simulation 
may  be  to  examine  a  course  of  action  for  possible  flaws.  If  that  search  is  truncated  because 
of  time  constraints,  the  decision  maker  may  proceed  with  an  incompletely  evaluated  course 

of  action. 

Mental  simulation  is  related  to  a  variety  of  constructs  in  psychology  including  mental 
imagery,  progressive  deepening,  and  mental  models,  but  has  unique  features  as  well.  These 
include  such  essential  functions  as:  evaluating  situation  assessment,  learning  about  the 
dynamics  of  a  situation,  evaluating  courses  of  actions  and  improving  upon  them,  and 
generating  expectancies. 

Prior  to  the  work  completed  under  this  contract,  the  most  explicit  account  of  mental 
simulation  that  we  have  been  able  to  locate  is  that  of  Jungermann  and  Thiiring  (1987).  Their 
four  stage  model  is  depicted  in  Figure  3.  Their  model,  though  a  good  first  step,  is  so 
general  that  its  explanatory  or  predictive  utility  is  very  limited.  The  several  studies  of 
mental  simulation  we  have  conducted  have  led  to  a  more  fully  developed  descriptive  account 
of  mental  simulation  and  its  role  in  decision  making. 

Like  much  of  decision  research,  mental  simulation  has  generally  been  studied  under 
specified  conditions,  using  well-defined  tasks.  We  know  litde  about  the  kinds  of  mental 
simulations  people  engage  in  or  the  purposes  they  serve  in  people’s  day-to-day  functioning. 
Our  approach  to  the  study  of  decision  making  has  consistently  emphasized  the  value  of 
grounding  research  in  the  tasks  and  settings  in  which  the  activity  occurs.  It  made  sense  to  us 
to  takft  a  look  at  how  people  use  mental  simulation  in  their  everyday  lives,  and  to  work  from 
there  to  develop  a  model  that  describes  those  activities. 

We  undertook  an  ethnographic  study  of  mental  simulation,  designed  to  offer  an  initial, 
exploratory  look  at  the  role  of  mental  simulation  in  planning,  decision  making,  and  problem 
solving.  For  the  study,  we  developed  a  database  of  104  mental  simulation  descriptions— 
incidents  which  decision  makers  report  relying  on  some  form  of  mental  simulation  for 
problem  solving  and  decision  making. 

Within  our  sample  of  mental  simulation  incidents,  we  identified  four  primary 
functions  served  by  the  simulations:  generate  and  implement  a  plan;  explain  a  phenomenon. 
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discover  and  explore  models  of  a  problem  or  process;  and  inspect  and  evaluate  a  course  of 
action.  Category  descriptions  follow:'* 


Definition  of  domain  and  task 


emulation  of 
the  mental  model 


selection  of 
inferences 


Figure  3.  The  cognitive  process  of  scenario  construction 

(From  "The  use  of  causal  knowledge  for  inferential  reasoning’  by  H.  Jungermann  and  M.  ThUring,  1987,  NATO 
ASI  Series.  F35.  131-146.)  _ _ _ _ _ 


Generate  a  Plan 

This  category  includes  mental  simulations  wherein  the  decision  maker  attempts  to 
envision  a  course  of  action.  The  mental  simulation  is  used  to  create  a  plan,  generate 
information,  and  develop  expectancies  about  events  that  are  likely  to  occur  in  the  mentally- 
simulated  situation.  The  simulations  are  typically  convergent,  focused  on  identifying  action 
sequences,  and  time-bound.  They  typically  take  a  current  situation  or  event  and  project  ite 
development  over  time,  from  the  present  into  the  future.  The  simulations  allow  the  decision 
maker  to  anticipate  the  "look  and  feel"  of  ensuing  events,  and  to  adequately  prepare  for 


‘‘Two  coders  independently  categorized  each  item  in  the  database  into  the  four  categories. 
Coding  reliability  was  assessed  using  kappa,  a  chance-corrected  index  of  agreement  (Fleiss, 
1981).  Overall  kappa  across  all  four  categories  was  .47,  where  .40  is  deemed  acceptable  and 
.75  is  deemed  very  good.  The  informal  nature  of  the  incident  accounts  made  it  hard  to  judge 
category  membership  in  many  cases. 
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them.  Once  actually  in  the  situation,  the  decision  maker  uses  these  simulation-based 
expectancies  to  evaluate  his/her  assessment  of  the  situation.  As  the  situation  unfolds, 
discrepancies  between  what  the  decision  maker  expected  and  what  s/he  observes  can  serve  as 
a  flag  that  the  original  situation  assessment  was  off-base,  or  that  critical  components  have 
changed  and  the  situation  assessment  needs  to  be  re-evaluated. 

Mental  simulations  that  serve  a  planning/anticipation  function  appear  to  be  particularly 
important  where  time  limits  are  severe  and  risks  high.  An  experienced  decision  maker 
knows  that  once  in  the  situation  there  will  not  be  time  to  analyze  and  evaluate,  and  so  uses 
mental  simulation  to  perform  those  functions  ahead  of  time. 

Explain 

This  category  includes  mental  simulations  conducted  to  help  the  decision  maker 
understand  how  and  why  an  event  could  have  occurred.  They  involve  the  se^ch  for 
plausible  explanations  and  are  typically  focused  backward  in  time.  The  decision  maker 
attempts  to  generate  a  feasible  account  of  how  an  end  state  could  have  arisen,  or  the  factors 
that  could  account  for  an  anomaly  that  has  occurred.  Inability  to  generate  a  plausible 
explanation  can  cause  the  decision  maker  to  abandon  the  mental  simulation  altogether  and  to 
reexamine  other  aspects  of  his  or  her  thinking. 

Mod  el /Discover 

This  type  of  mental  simulation  is  used  to  deepen  the  decision  maker  s  understanding 
of  some  process  or  problem.  The  simulations  often  have  a  musing,  expansive  quality.^ 
goal  of  these  simulations  is  to  create  a  hypothetical  model  or  system,  and  to  "observe  it  in 
action.  They  often  involve  examination  of  causal  factors  and  the  interactions  among  them, 
and  they  tend  to  be  multifaceted.  The  simulation  typically  proceeds  iteratively.  Knowledge 
gained  from  each  run  of  the  simulation  enhances  subsequent  simulations  of  the  same  problem 
or  process. 

Tnspect/Evaluate 

This  category  includes  mental  simulations  that  allow  a  decision  maker  to  evoke  and 
assess  the  impact  of  a  specific  action  or  sequence  of  events,  including  the  flaws  or  risks 
associated  with  it.  The  decision  maker  asks,  "WUl  this  work?"  or  "What  could  go  wrong  at 
this  point?" 


Evaluation  simulations  often  involve  a  directed  search  for  specific  problems,  and  an 
assessment  of  probable  outcomes  that  includes  an  affective  component  (satisfaction  vs. 
disappointment).  The  affective  response  evoked  by  the  simulation  guides  the  person’s 
decision  whether  to  actually  implement  the  course  of  action  envisioned.  The  person  uses 
experience  to  detect  flaws  and  warning  signs  as  the  simulation  unfolds.  Analogies,  rules  of 
thumb,  etc.  enter  into  the  detection  of  problems.  These  flaws  fall  into  other  criteria 
suggested  by  Robinson  and  Hawpe  (1986):  consistency  and  plausibility.  In  addition  to  taking 
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consistency  and  plausibiUty  into  account,  the  inspection  will  cover  potential  flaws  in  the 
course  of  action. 

If  the  simulation  satisfies  these  tests,  the  person  c^  proceed.  If  it  fails,  the  person 
may  be  able  to  modify  it.  In  this  way,  the  course  of  action  is  improved.  In  our  sample,  we 
found  that  the  decision  makers  tried  to  find  a  workable  scenario,  and  often  took  the  Ume  to 
evaluate  it  for  flaws.  Sometimes  they  tried  to  optimize  their  choice,  but  there  were  cases 
where  they  just  tried  to  confirm  that  a  course  of  action  would  not  lead  to  trouble. 

Application  of  the  Four  Categories  of  Mental  Simulation 

We  found  19  cases  in  which  the  simulation  was  used  to  generate  a  course  of  action  or 
to  generate  predictions  and  estimates,  36  cases  in  which  the  course  of  action  was  ev^uated 
and  inspected  for  risk,  11  cases  in  which  a  model  was  derived,  and  13  cases  in  which  the 
simulation  was  used  to  explain  a  situation.  There  were  an  additional  25  cases  that  were 
judged  not  to  meet  the  criteria  for  mental  simulation. 

For  the  cases  that  could  be  judged,  the  simulation  was  primarily  visual  (56  were 
judged  to  be  visual  and  23  were  judged  not  to  be  visual).  Regarding  time  stress,  26  were 
judged  to  have  high  stress  and  37  to  have  low  stress,  with  the  category  not  applicable  to  16 
cases.  Risk  was  generally  low,  with  16  cases  involving  high  risk  and  61  involving  low  nsk, 
and  the  category  not  applicable  in  2  cases. 

This  preliminary  study  of  mental  simulation  has  served  as  the  basis  for  two  additional 
investigations  of  planning  and  risk  inspection  simulations.  These  will  be  described  later  in 
this  section. 


The  ethnographic  study  was  also  extremely  useful  as  a  first  step  towards  developing  a 
conceptual  model  of  simulation  that  describes  the  role  of  simulation  in  naturalistic  decision 
making  and  problem  solving.  A  description  of  the  model  follows. 

We  have  derived  a  general  account  of  mental  simulation,  which  is  presented  in  Figure  4. 
A  person  uses  mental  simulation  to  accomplish  some  goal,  so  the  first  function  is  to  understand 
the  need  motivating  the  activity.  For  a  given  need,  the  next  step  depicted  in  Figure  4  is  to 
specify  the  components  of  the  simulation.  The  person  needs  to  determine  whether  the  task  is  to 
interpolate  between  a  known  initial  condition  and  a  known  terminal  condition,  to  extrapo  ate 
forward  from  a  known  initial  condition  (as  in  predicting  the  future),  or  to  extrapolate  backw^d 
from  a  known  terminal  state,  as  in  trying  to  explain  a  situation,  ^e  person  needs  to  identify 
the  casual  factors  that  will  be  used  in  the  simulation.  Our  data  did  not  show  that  this  step  was 
done  deliberately,  e.g.,  choosing  these  factors  from  a  larger  set,  writing  them  down,  etc. 
Rather,  the  decision  makers  used  their  experience  to  identify  the  most  relevant  factors  to  work 
with,  and  appeared  to  rely  on  metacognitive  skills  to  anticipate  how  many  factors  to  consider. 
With  too  few  factors,  it  is  hard  to  formulate  an  interesting  simulation,  whereas  too  many  ^e 
hard  to  keep  track  of.  Skill  is  needed  to  anticipate  what  wUl  be  a  useful  level  of  complexity. 
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As  an  example,  we  asked  people  to  generate  a  mental  simulation  of  the  economic  events 
in  Poland,  following  the  shock  transformation  into  a  market  economy  in  January,  1990.  The 
data  were’ gathered  in  February/March  1990.  We  collected  mental  simulation  data  from  one 
person  who  had  a  Ph.D.  in  economics,  and  had  come  to  the  U.S.  from  Poland  at  the  age  of  20. 
He  generated  a  mental  simulation  with  three  factors  (inflation  rate,  exchange  rate,  and 
unemployment),  and  showed  how  these  would  fluctuate  and  interact  with  each  other  during  the 
coming  year.  He  knew  the  current  rates,  and  so  could  generate  specific  estimates  for  all  three 
parameters,  quarter  by  quarter.  In  contrast,  two  other  people  with  less  experience  were  unable 
to  construct  useful  simulations.  They  each  considered  unemployment  and  inflation,  forgot  about 
exchange  rate,  were  not  able  to  see  how  the  two  factors  of  unemployment  and  mflation  would 
affect  each  other,  and  did  not  know  the  current  levels  of  inflation  and  unemployment.  Needless 
to  say,  they  were  unable  to  "run"  their  simulation  in  order  to  generate  any  estimates  at  all. 
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The  third  step  depicted  in  Figure  4  is  to  assemble  the  simulation.  In  our  database,  many 
simulations  consisted  of  transitions  from  one  state  to  another.  The  purpose  of  the  simulation 
was  to  examine  the  nature  of  these  transitions.  It  appears  that  people  naturally  Umit  the  number 
of  factors  and  transitional  states  they  employ  in  a  simulation.  In  our  study,  the  median  number 
of  transitions  was  about  six,  and  the  median  of  causal  factors  that  varied  was  three,  with  a 
range  of  one  to  six.  So  we  have  represented  mental  simulation  as  an  action  sequence  with  six 
progressions.  We  also  hypothesize  that  most  mental  simulations  will  center  around  three 
operators  or  causal  factors. 

Sometimes,  the  initial  state  will  be  given,  and  the  person  will  have  to  extrapolate  to  an 
end  state;  at  other  times  both  the  initial  and  terminal  states  will  be  known,  and  the  task  will  be 
to  interpolate--to  bridge  the  gap. 


We  have  depicted  the  transitions  from  one  state  to  another  as  discrete,  rather  than 
continuous.  Therefore,  "configuring  the  simulation"  in  one’s  h^d  involves  selecting  a  smaU 
number  of  variables  and  manipulating  these  over  several  transitions— like  successive  frarnes  in 
an  animated  sequence.  Not  all  mental  simulations  are  experienced  as  a  series  of  discontinuous 
action  states;  many  appear  to  run  in  a  continuous  action  flow.  But  in  these  instances  as  well, 
the  number  of  factors  and  the  span  of  the  simulating  are  constrained. 

The  internal  coherence  of  the  simulation  is  judged  using  some  of  the  criteria  listed  by 
Robinson  and  Hawpe  (1986)  for  evaluating  stories:  coherence;  applicabiUty;  and 
completeness/selectivity.  Pennington  and  Hastie  (in  press)  have  suggested  that  stones  are  more 
acceptable  as  a  function  of  completeness,  consistency,  plausibility,  and  uniqueness.  In  other 
words,  the  simulation  has  to  make  sense,  it  has  to  address  the  important  features  of  the  situation 
but  not  drown  in  excessive  detail,  and  it  has  to  apply  to  the  goals.  If  the  simulation  fails  these 
tests,  it  will  be  necessary  to  try  again  or  to  rethink  the  goals.  If  it  succeeds,  then  the  person 
runs  the  simulation  and  uses  the  information  gained  from  it. 

To  illustrate  the  model,  we  offer  an  example.  The  task  is  to  have  one  person  (working 
alone)  raise  a  five-ton  truck  to  a  height  of  four  feet  off  the  ground,  supported  in  its  center  by  a 
column  of  concrete  blocks.  The  available  equipment  includes  a  jack,  and  a  large  pile  of 
concrete  blocks.  How  might  one  accomplish  the  task? 

One  approach  would  be  to  use  the  jack  to  lift  one  tire,  placing  blocks  underneath  it,  then 
lift  another  tire,  and  so  forth  until  the  truck  is  in  the  air.  Then  one  could  begin  to  build  a 
column  of  blocks  beneath  the  truck,  raising  it  gradually  and  constructing  a  platform  of  blocks 
under  the  center,  releasing  the  jack  so  the  truck  is  resting  on  the  center  column  of  blocks,  and 
removing  the  blocks  from  each  of  the  tires.  In  this  way,  one  could  mentally  play  out  an  action 
sequence,  examine  it,  and  revise  it  before  actually  attempting  the  task. 

Let  us  use  the  truck  example  to  contrast  the  different  needs  that  can  be  served  by  a 
mental  simulation  (see  Figure  4). 
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Generate/predict'.  The  task  here  is  to  assemble  an  action  sequence.  In  the 
truck  example,  the  challenge  is  primarily  to  fmd  a  course  of  action  that  will  do 
the  job.  This  category  includes  a  sub-category  of  predicting  events  and  data. 

The  task  is  to  run  the  action  sequence,  to  determine  estimates  of  different 
parameters  (e.g.,  time  needed  to  raise  the  truck).  As  the  action  sequence  is  run, 
it  is  possible  to  generate  estimates  of  the  time  required  for  each  of  the  segments. 

Explain:  The  task  is  to  understand  how  an  end  state  could  have  arisen. 
Thus,  if  you  saw  a  truck  resting  on  a  pile  of  bricks,  and  you  were  told  that  one 
person,  acting  alone,  had  managed  the  feat,  you  might  try  to  pnerate  an  action 
sequence  to  account  for  how  it  was  done.  The  process  is  similar  to  generating  a 
course  of  action,  except  that  the  failure  to  assemble  a  simulation  is  taken  as 
evidence  against  the  proposed  explanation. 

Mndd/discnver.  The  interplay  of  causal  factors  is  unknown,  and  the  task 
is  to  understand  it.  For  example,  one  might  imagine  tilting  the  truck  far  back,  or 
at  extreme  angles  from  side  to  side,  to  try  to  envision  its  center  of  gravity  and  its 
stability.  The  goal  here  is  to  create  a  hypothetical  model,  typically  in  order  to 
deepen  one’s  understanding  of  some  domain.  The  simulation  is  examined 
iteratively  and  the  knowledge  gained  is  fed  back  into  the  simulation. 

Evaluate:  The  task  is  to  judge  the  adequacy  of  the  expected  outcome.  In 
the  example  of  lifting  a  truck,  the  evaluation  would  hinge  on  the  adequacy  of  the 
bridge  between  initial  and  terminal  states.  In  open-ended  tasks,  the  evaluation 
would  focus  on  the  quality  of  the  terminal  state.  In  our  example,  the  person 
would  look  at  the  process  and  determine  whether  it  seemed  workable  or  not. 

There  is  another,  more  specific  type  of  evaluation,  in  which  the  person 
examines  the  simulation  for  ways  in  which  it  might  fail.  The  person  draws  on 
his  or  her  experience  to  detect  warning  signs  as  the  imagined  action  sequence 
unfolds.  Analogies,  rules  of  thumb,  etc.,  enter  into  the  detection  of  potential 
problems.  If  the  simulation  appears  satisfactory,  the  person  can  proceed.  If  it 
fails,  the  person  may  be  able  to  modify  it.  For  instance,  the  pressure  of  jacking 
up  one  comer  of  the  truck  may  produce  lateral  force  on  the  others,  causing  the 
truck  to  topple.  The  operator  may  realize  that  a  single  column  of  blocks  won’t 
be  sufficiently  stable-what  is  needed  is  a  broader  platform,  along  with  chocks  to 
keep  the  tires  from  shifting.  In  this  way,  the  course  of  action  gets  improved. 

In  our  simulation  database,  we  found  that  the  decision  makers  tried  to  find 
a  workable  scenario,  but  were  usually  careful  to  evaluate  it  for  flaws.  Sometimes 
they  tried  to  optimize  their  choice,  but  there  were  cases  where  they  just  tried  to 
confirm  that  a  course  of  action  would  not  lead  to  trouble.  We  suspect  that  in 
many  operational  environments,  use  of  simulation  to  inspect  a  plan  for  flaws  is 
common  if  and  when  time  permits. 
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The  inspection  of  the  simulation  can  also  help  a  person  prepare  for  action. 

The  task  here  is  to  find  difficult  steps  and  prepare  for  them.  The  person  uses 
experience  to  detect  warning  signs  as  the  action  sequence  unfolds.  Returning  to 
our  truck  example,  a  person  might  look  at  the  course  of  action  and  realize  s/he 
isn’t  sure  when  to  stop  jacking  up  one  of  the  comers,  before  the  truck  becomes 
unstable.  The  person  may  realize  it  is  important  to  concentrate  on  visualizing 
early  signs  of  imbalance. 

How  might  we  put  this  account  of  mental  simulation  into  practice?  Means,  Salas, 
Crandall,  and  Jacobs  (in  press)  have  argued  that  attempts  to  train  decision  skills  have  largely 
been  ineffective,  with  the  exception  of  training  metacognition.  This  is  consistent  with  our 
understanding  of  naturalistic  decision  making,  since  most  prescriptive  strategies,  such  as 
MAUA,  are  inappropriate  for  the  ill-defined  and  time-pressured  tasks  usually  encountered. 
Further,  it  makes  little  sense  to  train  people  to  use  naturahstic  strategies  such  as  recognitional 
decision  making,  since  these  models  describe  the  way  people  ordinarily  handle  decision  tasks. 

Nevertheless,  there  is  one  candidate  for  training  that  has  interested  us,  and  that  is  mental 
simulation.  The  work  of  Klein  and  Crandall  (in  preparation)  suggested  a  few  generic 
difficulties  that  people  have  with  mental  simulation,  including  overconfidence  and  failure  to 
conduct  risk  assessments.  We,  therefore,  attempted  to  develop  a  method  for  training  better 
mental  simulation  practices. 


Kyne,  Militello,  and  Klein  (in  preparation)  performed  two  studies  on  mental  simulation 
strategies.  The  goal  of  the  first  study  was  to  explore  the  role  of  mental  simulation  in  evaluating 
a  plan  for  errors.  We  have  used  the  term  risk  inspection  to  describe  the  process  where  one 
mentally  walks  through  a  plan  before  it  has  been  carried  out,  in  order  to  find  flaws  or 
weaknesses  in  the  plan.  We  are  working  to  develop  a  mental  simulation  training  procedure 
which  we  believe  will  enhance  planning  quality  through  the  use  of  risk  inspection.  This 
procedure  can  be  considered  a  "pre-mortem"  in  which  the  possible  causes  of  failure  are 
considered  in  advance  of  the  event  rather  than  after  it.  It  both  motivates  a  person  to  perform  a 
risk  inspection  and  inspires  a  person  to  be  less  invested  in  his/her  original  plan. 

The  study  focused  on  changing  a  person’s  commitment  to  his/her  original  plan.  We 
asked  participants  to  plan  a  course  of  action,  and  to  then  conduct  mental  simulations  of  how  the 
plan  might  fail.  We  attempted  to  change  commitment  to  the  course  of  action  in  the  following 
way;  during  the  risk  inspection  phase,  we  asked  participants  to  take  the  role  of  a  devil  s 
advocate,  with  the  power  to  thwart  a  plan.  Their  task  was  to  implement  Murphy’s  law  by 
going  back  over  the  original  plan  searching  for  what  could  go  wrong  and  describing  those 
weaknesses  for  us. 

The  participants  were  41  Psychology  majors  at  Wright  State  University,  Dayton,  OH. 
The  initial  simulation  task  was  to  develop  a  plan  for  getting  to  the  Dayton  airport  in  time  to 
greet  an  overseas  visitor. 

To  test  whether  a  person  was  less  committed  to  his  or  her  original  plan,  we  asked 
participants  to  rate  their  confidence  in  the  plan  on  a  seven-point  scale.  The  mean  confidence 
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rating  was  4.83.  The  participants  then  attempted  to  find  ways  in  which  the  plan  would  fail. 
Confidence  in  the  plan  was  again  rated,  and  the  mean  fell  to  3.83.  To  test  whether  the  decline 
was  significant,  we  computed  difference  scores  for  each  subject  and  compared  the  mean 
discrepancy  to  zero.  The  t-test  revealed  a  significant  change  (t=4.20;  df=40,  p<  .01).  The 
data  support  the  use  of  this  approach  for  alerting  decision  makers  to  potentid  flaws.  Moreover, 
the  method  appears  applicable  to  many  situations,  since  a  person  could  use  it  on  a  plan  without 
knowing  in  advance  where  the  problematic  assumptions  were. 

The  second  study  was  a  more  systematic  attempt  to  contrast  a  pre^ mortem  condition  with 
control  groups  that  either  reviewed  the  plan  or  just  relaxed  during  the  extra  time.  This  time  we 
looked  at  both  changes  in  confidence  and  changes  in  plan  quality.  The  pre-mortem  condition  in 
this  study  was  given  a  sUghtly  different  instruction-they  were  asked  to  imagine  that  their  plan 
had  been  carried  out,  and  that  the  plan  had  failed.  They  were  challenged  to  think  of  plausible 
reasons  why  this  failure  might  have  occurred,  and  to  write  down  as  many  good  reasons  as  they 
could. 


Participants  were  36  first  through  fifth  year  graduate  students  at  Wright  State  University, 
Ohio  State  University,  and  Bowling  Green  State  University,  all  in  Ohio. 

Participants  were  presented  a  scenario  in  which  their  academic  advisor  had  agreed  to 
present  the  results  of  a  study  at  a  professional  conference.  Descriptions  of  certain  constraints 
such  as  time,  money,  and  other  appropriate  resources  were  included  in  the  scenario. 

Participants  were  asked  to  generate  a  plan  to  carry  out  the  experiment  within  the  given 
constraints.  Researchers  then  intervened  with  one  of  three  instructional  sets; 

•  Pre-mortem  -  participants  were  instructed  to  imagine  their  plan  had  failed  and  to 
identify  reasons  for  the  failure  and  vulnerabilities  to  their  plan. 

•  Review  -  participants  were  instructed  to  review  their  plan. 

•  Control  -  participants  were  instructed  to  "just  relax"  and  no  mention  was  made  of 
their  plan. 

Participants  were  then  asked  to  revise  their  plans. 

We  asked  participants  to  rate  confidence  in  their  plan  and  the  probability  of  success  three 
times  during  the  experiment.  We  asked  the  participants  to  complete  the  following  surveys: 

-  Rate  confidence  and  probability  of  success  in  their  original  plan  immediately  after 
completing  it. 

—  Rate  confidence  and  probability  of  success  in  their  original  plan  after  intervention. 

-  Rate  confidence  and  probability  of  success  in  their  revised  plan. 
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Our  results  replicate  our  findings  from  the  first  risk  inspection  traming  study.  When 
tested  against  zero,  there  was  significant  decline  in  subjective  ratings  of  probability  of  success 
(t=2.32;  df=ll,  p<.02)  and  confidence  from  survey  one  to  survey  two  (t=2.01;  df=ll, 
p<  .05)  for  the  pre-mortem  group.  The  same  tests  failed  to  find  significant  decreases  in 
confidence  or  probability  rating  for  the  Review  or  the  Control  group.  When  considering  that 
the  pre-mortem  strategy  was  described  in  seven  minutes,  and  no  practice  was  provided,  these 

are  powerful  results. 

In  addition  to  examining  changes  in  confidence  level,  we  also  investigated  changes  in 
plan  quality.  We  evaluated  the  quality  of  the  plans  generated  before  and  after  the  treatment 
condition,  for  each  of  the  three  groups.  Two  raters  reviewed  all  of  the  plans  and  provided 
evaluations,  without  knowing  which  of  the  three  conditions  the  plans  fell  into.  We  expected 
that  the  pre-mortem  condition  would  show  a  significant  increase  in  pl^  quality,  and  it  did, 
however,  plan  quality  increased  significantly  in  all  of  the  three  conditions,  F(343)— 32.149,  p<  .001. 
Moreover,  there  was  no  interaction  between  treatment  condition  and  plan  q^uality.  Therefore, 
it  appears  as  if  the  pre-mortem  procedure  was  not  significantly  more  effective  in  improving 
plan  quality,  even  though  it  reduced  confidence  levels  for  the  original  plan. 

We  then  examined  the  relationship  between  plan  quality  and  shifts  in  confidence  level 
(r  =  -.  10).  Specifically,  we  wanted  to  know  if  those  subjects  whose  confidence  went  down 
after  the  treatment  condition  also  showed  the  most  improvement  in  plan  quality ,  regardless  of 
which  condition  they  were  in.  The  data  showed  no  such  relationship.  Changes  in  confidence 
were  not  tied  to  changes  in  plan  quality.  These  findings  suggest  that  for  this  task,  beliefs 
and  behaviors  are  not  closely  related.  It  should  be  noted  that  the  two  raters  showed  only 
moderate  levels  of  agreement  with  each  other  on  total  plan  quality  (r  =  .692  and  .611)  and 
were  therefore  measuring  somewhat  different  features  of  the  plans,  so  it  will  be  necessary  to 
perform  additional  studies  to  verify  these  findings.  Nevertheless,  the  issue  is  raised  of  how 
to  use  mental  simulation  and  the  pre-mortem  method  to  directly  improve  plan  quality. 

Before  leaving  this  section,  it  may  be  worthwhile  to  note  one  additional  point.  'Die 
concept  of  mental  models  (e.g. ,  Rouse  &  Morris,  1986)  has  often  been  cited  as  explaining 
different  phenomena,  but  suffers  from  lack  of  precision.  Rouse  and  Morris  caution  that  too 
often,  "mental  models"  refers  to  everything  in  a  person’s  head,  and  that  this  is  too  inclusive 
and  uninterpretable  to  have  much  value.  In  developing  our  account  of  mental  simulation,  we 
realized  that  a  mental  simulation  was  a  plausible  causal  chain  of  events,  involving  a  small  set 
of  active  variables.  In  other  words,  a  mental  simulation  was  a  story  that  could  be  judged  for 
plausibility.  There  are  many  different  types  of  models,  as  discussed  above:  spatial, 
statistical,  physical,  and  so  forth.  Causal  models,  e.g.,  explanations  of  the  way  a  certain 
type  of  cancer  grows,  are  basically  stories.  Perhaps  we  can  consider  certain  types  of  mental 
models  in  a  similar  fashion,  as  stories.  A  person’s  mental  model  of  how  a  piece  of  software 
is  operating  is  a  story  of  which  files  are  being  created  and  accessed,  and  how  this  is 
happening.  As  such,  it  should  be  possible  to  describe  the  mental  model  as  a  story,  and  to 
evaluate  it  using  the  criteria  used  to  assess  the  plausibility  of  stories.  This  strategy  may  be 
helpful  in  giving  more  shape  to  the  concept  of  mental  models. 
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Situation  Assessment 


In  its  earlier  forms,  the  RPD  model  described  four  variables  that  were  contained  in  a 
situation  assessment  (goals,  cues,  expectancies,  and  courses  of  action),  but  the  model  had 
nothing  to  say  about  the  way  a  situation  assessment  was  selected.  We  now  have  elaborated 
this  part  of  the  model.  We  can  Ust  several  strategies  for  arriving  at  a  situation  assessment. 
These  include  nonanalytical  recognition,  feature  matching,  story  building  ,  analogical 
reasoning,  and  belief  updating  (Einhom  &  Hogarth,  1987),  along  with  more  analytical 
methods  such  as  Bayesian  statistics. 

The  variations  of  the  RPD  model  are  shown  in  Figures  5-7.  The  simplest  path 
(Figure  5)  is  for  the  person  to  directly  recognize  a  situation  as  typical,  e.g.,  an  instance  of  a 
prototype  or  as  a  direct  analogue  to  a  previous  experience.  This  direct  recognition  can 
involve  a  holistic  process  such  as  would  be  involved  in  holographic  memory  or  neural  nets. 
Alternatively,  the  recognition  can  depend  on  feature  matching. 

Sometimes  the  decision  maker  will  evaluate  an  option  by  mentally  simulating  what 
might  happen  if  the  option  was  implemented,  to  see  if  it  needs  to  be  modified.  This  is 
shown  in  Figure  6. 

The  most  complex  version  of  the  RPD  model,  as  presented  in  Figure  7,  can  also 
apply  to  cases  where  a  decision  maker  must  select  between  several  alternative  situation 
assessments.  When  faced  with  such  a  decision,  as  in  diagnosing  what  is  wrong  with  a  car 
engine,  or  whether  a  defendant  is  guilty,  then  the  person  must  reassess  the  situation.  If  the 
original  identification  was  based  on  features,  then  a  feature  matching  process  may  suffice. 
More  often,  the  assessment  is  in  terms  of  a  story,  or  causal  account  linking  different  pieces 
of  evidence.  Here,  the  decision  makers  will  evaluate  the  alternative  stories  in  terms  of 
plausibility,  consistency,  and  so  on  (see  Klein  &  CrandaU,  in  preparation;  Pennington  &, 
Hastie,  in  press). 


Figure  5.  Simple  match. 
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plausible  causal  chain  that  links  up  the  different  observations;  this  is  the  story  model 
presented  by  Pennington  and  Hastie  (in  press),  and  it  also  corresponds  to  the  form  of  mental 
simulation  described  above  as  explaining  an  event. 
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(  Experience  Situation) 


Figure  6.  Developing  a  course  of  action. 


Figure?.  Complex  RPD  strategy. 


One  immediate  question  is  whether  the  processes  in  Figure  7  appropriately  fit  within  a 
model  of  recognitional  decision  making.  The  construction  and  evaluation  of  stories  is  hardly 
a  recognitional  process,  and  the  deliberate  use  of  feature  matching  likewise  does  not  count  as 
recognition.  Therefore,  the  elaborated  view  of  situation  assessment  may  be  more 
appropriately  considered  a  companion  model  to  the  RPD  strategy. 
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There  are  also  arguments  for  incorporating  these  aspects  of  situation  assessment  into 
the  RPD  model,  as  we  have  done  in  Figure  7 .  The  rationale  of  the  model  is  to  show  how 
experienced  decision  makers  can  directly  recognize  a  course  of  action,  once  a  situation 
assessment  is  arrived  at,  thereby  avoiding  the  need  for  concurrent  evaluation  of  options. 
Figure  7  is  consistent  with  this  goal.  Furthermore,  the  processes  by  which  multiple  situation 
assessments  are  generated,  e.g.,  holistic  matches  and  feature  matches,  are  recognitional 
processes  that  simply  happen  to  generate  more  than  one  match.  Finally ,  the  means  of 
selecting  a  situation  assessment,  whether  feature  matching  or  mental  simulation,  are  already 
processes  contained  within  the  RPD  model.  Thus,  the  RPD  model  already  contains  the 
components  needed  for  Figure  7,  so  it  does  not  seem  to  be  too  great  a  stretch  to  incorporate 
these  within  the  model.  We  must  be  careful  not  to  expand  the  model  to  include  every  aspect 
of  NDM,  or  the  model  wUl  lose  its  integrity,  but  in  this  case  it  appears  that  we  have 
remained  within  the  general  parameters  already  set  forth  by  the  model. 

There  are  other  accounts  of  how  situation  assessment  is  updated  that  do  not  appear  in 
Figure  7.  These  are  the  use  of  Bayesian  statistics  as  well  as  the  belief  updating  model  of 
Einhom  and  Hogarth  (1987). 

Other  suggestions  have  been  made  to  incorporate  processes  such  as  analogical 
reasoning,  representativeness,  and  availability  as  mechanisms  for  informing  situation 
assessment.  Clearly,  these  are  all  important  cognitive  processes  for  the  RPD  model. 
However,  we  have  chosen  not  to  incorporate  them  into  the  model  because  they  are  generic 
cognitive  processes;  if  we  incorporate  all  the  relevant  cognitive  processes  (e.g.,  memory 
retrieval,  inferential  activities,  pattern  matching,  motivation,  and  so  on),  we  would  wind  up 
with  a  model  of  all  the  topics  included  in  textbooks  on  cognitive  psychology. 

The  question  then  arises;  what  kind  of  a  model  is  the  RPD  model?  There  are 
different  types  of  models,  including  statistical  models,  spatial  models,  physical  models,  and 
causal  models.  The  RPD  is  a  causal  model.  It  was  developed  through  observations  of  real- 
life  decision  makers;  thus,  its  roots  are  in  a  descriptive,  not  a  prescriptive  decision-rnaking 
approach~it  is  an  account  of  the  major  processes  responsible  for  carrying  out  a  certain  type 
of  strategy.  It  omits  many  basic  mechanisms  (e.g.,  the  mechanisms  of  drawing  inferences, 
the  mechanisms  of  retrieval),  just  as  any  account  leaves  out  processes  that  do  not  distinguish 
it  from  alternative  accounts.  The  details  of  the  RPD  model  are  intended  to  draw  the 
distinction  between  RPD  and  MAUA  as  sharply  as  possible. 

RPD;  Testing  the  Model 

The  RPD  model  assumes  that  experienced  decision  makers  can  generate  feasible 
options  as  the  first  ones  considered.  This  is  a  strong  and  critical  assumption.  In  a  domain 
where  there  are  large  sets  of  options,  and  where  most  of  the  options  are  unsatisfactory,  the 
use  of  an  RPD  strategy  would  be  counter-productive. 

The  assumption  of  directed  rather  than  random  generation  of  options  stands  in 
contrast  to  classical  decision  theories.  We  are  not  claiming  that  the  classic^  theories  assert 
that  option  generation  will  be  random;  the  idea  of  a  directed  option  generation  is  not 
inconsistent  with  these  theories.  Nevertheless,  the  dilemma  normative  models  are  trying  to 
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resolve  does  seem  to  be  tied  to  random  option  generation.  The  dilemma  is  how  decision 
makers  faced  with  large  sets  of  options  can  focus  down  to  find  a  good  one  to  select.  Under 
time  pressure,  it  is  impractical  to  consider  a  large  set  of  options.  We  would  expect  decision 
making  to  become  ineffective,  and  yet  we  know  that  is  not  the  case.  Perhaps  people  are  able 
to  select  a  small  group  of  promising  options  from  the  initial  large  set,  and  are  thereby  able  to 
handle  the  time  pressure.  Yet  even  this  hypthesis  runs  into  difficulty  since  it  is  not  clear 
how  a  person  would  do  so  without  first  reviewing  the  entire  large  set. 

The  advantage  of  using  a  recognitional  strategy  is  that  it  provides  a  reasonable  option 
as  the  first  one  considered,  and  thus  avoids  the  dilemma  altogether.  There  is  no  need  to 
review  a  large  set  of  options,  or  even  to  whittle  this  set  down,  if  the  first  option  is  workable. 
Therefore,  the  RPD  model  predicts  that  if  subjects  are  asked  to  generate  options,  the  first 
ones  they  generate  will  be  acceptable  and  of  better  quality  than  the  average  member  of  the 
option  set. 

Klein,  Wolf,  Militello,  and  Zsambok  (in  preparation)  tested  this  hypothesis  using 
chess  as  the  domain  of  inquiry.  Four  chess  boards  were  selected  as  being  representative  of 
challenging  move  selection  requirements.  We  identified  all  of  the  legal  moves  in  each  board 
position,  and  obtained  objective  ratings  for  each  of  these  moves,  using  previously  published 
evaluations  prepared  by  panels  of  grandmasters. 

The  subjects  were  16  chess  players  recruited  from  the  Dayton,  Ohio  chess  club.  We 
tested  subjects  at  two  skill  levels  (as  established  by  their  standardized  chess  ratings).  The 
stronger  group  corresponded  to  "A"  level  strength,  and  the  weaker  group  corresponded  to 
"C"  level  strength.  For  purposes  of  interpretation,  the  strongest  players  are  grandmasters, 
then  masters,  then  experts,  then  classes  A,  B,  C,  and  D.  So,  the  stronger  players  were  very 
good  but  not  outstanding,  and  the  weaker  players  were  still  experienced  and  better  than  most 
players  in  the  general  population. 

The  study  consisted  of  four  trials  with  each  subject,  one  for  each  board  position.  The 
subjects  were  asked  to  generate  a  think-aloud  protocol  as  soon  as  the  chess  board  position 
was  shown  to  them,  in  which  they  articulated  each  move  they  identified.  Once  they  arrived 
at  a  decision  about  which  move  they  would  play,  they  were  told  they  could  stop  generating 
moves.  When  all  four  boards  were  completed,  the  subjects  were  asked  to  go  over  ^ch 
board  again  and  to  rate  the  quality  of  each  of  the  legal  moves.  The  reason  for  obtaining 
these  self-ratings  was  that  grandmaster  evaluations  could  be  misleading.  Suppose  there  is  a 
subtle  and  outstanding  move  according  to  the  grandmasters,  but  a  class  C  chess  player  does 
not  see  the  implications  of  the  move,  and  thinks  it  is  a  blunder .  For  the  purposes  of  this 
study  the  move  should  count  as  a  blunder,  because  the  subject  did  not  generate  what  s/he 
thought  was  a  strong  move.  The  rating  scale  used  ran  from  5  =  outstanding  move,  to 
1  =  blunder. 

The  results  strongly  supported  the  RPD  model.  Figure  8  shows  the  contrast  between 
the  distribution  of  all  legal  moves,  and  the  first  moves  generated.  If  subjects  were  randomly 
selecting  moves  from  the  set  of  legal  moves,  their  distributions  would  foUow  the  solid  line  in 
Figure  9,  and  most  moves  would  be  poor,  with  few  good  moves.  The  actual  distribution  of 
ratings  for  first  moves  considered,  shown  by  the  dashed  line,  ran  in  the  opposite  direction. 
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Few  first  moves  were  poor.  Actually,  few  were  lower  than  a  rating  of  three,  which  was 
assigned  to  acceptable  but  not  outstanding  moves.  Most  of  the  moves  were  rated  four  or 
five.  The  difference  between  the  actual  distribution  and  the  one  expected  from  random 
generation  was  highly  significant,  =  211.145,  p  <  .0001.  The  class  A  players  showed  a 
slightly  more  pronounced  difference  from  random  generation,  but  it  was  not  significantly 
different  from  the  Class  C  players. 


Figure  8.  Proportion  of  ratings. 


We  also  investigated  the  objective  quality  of  the  moves  generated,  and  these  data  are 
presented  in  Table  1.  The  panel  of  grandmasters  assigned  points  to  each  move  diat  they  felt 
was  worth  considering  in  a  position.  Most  of  the  legal  moves  were  given  no  points  at  all. 

We  used  these  data  to  assign  moves  into  two  classes:  those  worth  considering  (i.e.,  those 
receiving  any  points  at  all)  and  those  not  worth  considering.  The  ratio  was  about  5.1,  with 
only  20  of  the  124  legal  moves  on  the  four  boards  receiving  any  points.  If  subjects  were 
randomly  sampling  from  legal  moves,  this  ratio  should  have  held  for  the  moves  they 
generated.  In  fact,  the  ratio  of  moves  worth  considering  vs.  not  worth  considering  was  about  1:1, 
and  the  difference  from  random  generation  was  again  highly  significant,  =  113.9,  p  <  .0001. 
So  even  by  objective  standards,  the  subjects  were  doing  a  good  job  of  generating  acceptable 
moves  as  the  first  one  considered. 

We  also  noted  that  even  when  subjects  generated  a  poor  move  early  in  die  sequence, 
they  rapidly  assessed  it  as  being  of  low  quality  and  discarded  it,  so  there  was  litde  time 
penalty  for  a  serial  generation/evaluation  strategy  even  when  it  did  not  succeed  in  generating 
a  strong  move  initially. 
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Table  1 


Frequency  of  Subject-Generated  vs.  All  Legal  Moves  by  AcceptabUity  Level 


Acceptable  (Grand  Master 
points  were  awarded) 

Unacceptable  (No  Grand 
Master  points  were  awarded) 


Moves  generated  All  legal  moves 

by  the  subjects _ _ _ 


41  moves 

20  moves 

23  moves 

104  moves 

These  findings  bring  into  question  the  value  of  advising  decision  makers  to  generate  a 
large  set  of  options  in  order  to  be  sure  of  finding  a  good  one  (e.g,,  Gettys,  1983).  There 
will  be  some  conditions  under  which  this  advise  holds,  but  for  many  other  conditions, 
reasonably  experienced  decision  makers  can  make  more  profitable  use  of  their  time  by 
relying  on  their  ability  to  generate  acceptable  courses  of  action,  and  can  use  their  time  to 
conduct  more  careful  mental  simulations  to  evaluate  the  consequences  of  these  actions. 

The  data  also  suggest  that  we  must  be  careful  in  interpreting  studies  such  as  Payne, 
Bettman,  and  Johnson  (1988),  in  which  the  costs  and  benefits  of  different  dechion  strategies 
were  evaluated  assuming  random  generation  of  options.  If  option  generation  is  not  random, 
then  these  costs  and  benefits  would  likely  be  different. 

The  data  also  bear  on  the  link  between  skill  level  and  decision  strategy.  The  RPD 
model  makes  sense  at  skill  levels  where  feasible  options  can  be  recognized.  It  makes  less 
sense,  and  is  less  valuable,  for  people  at  lower  skill  levels  (although  it  is  not  clear  diat 
exhaustive  generation  and  selective  evaluation  will  work  any  better).  This  study  raises  the 
issue  of  the  benefits  and  costs  of  recognitional  versus  analytical  decision  strategies,  as  a 
function  of  experience  level. 

In  summary,  this  study  represents  the  first  empirical  test  of  the  RPD  model  in  which 
a  prediction  was  generated  and  rigorously  tested.  We  have  encountered  two  kinds  of 
criticisms  in  describing  the  results  of  this  study.  The  first  is  that  the  findings  were  obvious, 
since  no  one  really  believes  in  random  generation  of  options.  In  response  to  this  criticism, 
we  point  to  the  Payne  et  al.  (1988)  study  as  an  example  of  a  study  predicated  on  random 
generation.  The  second  criticism  is  that  the  results  were  not  conclusive  owing  to  the 
shortcomings  of  protocol  studies,  so  the  subjects  still  could  have  been  generating  options 
randomly,  but  censoring  themselves  during  the  self-report.  In  response  to  this  criticism,  we 
concur  that  protocol  analysis  caimot  fully  tap  cognitive  processes  of  the  subjects.  We  med  to 
make  sure  the  demand  characteristics  did  not  pressure  subjects  into  self-censorship,  and  we 
did  find  a  number  of  instances  in  which  blunders  were  generated  and  then  noticed.  It  is 
interesting  that  these  two  criticisms  stand  in  opposition  to  each  other-to  say  that  no  one 
believes  in  random  generation  is  the  opposite  of  saying  that  our  method  has  not  ruled  out 
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random  generation  so  it  is  stiU  a  possibiUty.  Our  own  assessment  is  that  the  prediction  based 
on  the  RPD  model  was  confirmed,  and  that  the  burden  of  proof  has  now  shifted  to 
proponents  of  random  generation  of  options. 

Option  Generation  and  Recognitional  Decision  Making 

In  this  section,  we  present  a  preliminary  description  of  how  decision  makers  generate 
options.  Despite  the  importance  of  option  generation  for  problem  solving  and  decision 
making,  there  has  been  surprisingly  little  written  about  the  mechanism  itself.  Random  option 
generation  is  an  approach  that  is  not  very  enlightening,  and,  based  on  the  findings  just 
described,  not  particularly  accurate.  Analogical  option  generation  is  an  attractive  idea,  since 
analogical  reasoning  captures  some  of  the  experience  base  needed  to  move  beyond  random 
option  generation,  but  even  analogical  models  are  unsatisfactory  because  it  is  rare  in  most 
domains  to  find  a  perfect  analogue.  There  will  always  be  some  differences  between  a  prior 
case  and  the  current  one,  as  well  as  conflict  between  different  possible  analogues. 

Furthermore,  models  of  analogical  reasoning  have  been  more  concerned  with  mapping 
analogues  to  different  cases,  as  opposed  to  describing  how  analogues  might  be  used  to 
generate  courses  of  action  in  a  new  situation.  Adelman  (personal  communication)  has 
recently  studied  factors  that  influence  which  options  are  generated,  but  has  not  addressed  the 
mechanism  of  option  generation.  We  need  to  go  back  to  the  work  of  Duncker  (1945)  and 
de  Groot  (1965/1978)  to  find  serious  attempts  to  explain  how  decision  makers  generate  a 
course  of  action.  The  account  we  present  below  is  influenced  by,  and  consistent  with,  the 
work  of  Duncker  and  de  Groot. 

In  the  course  of  conducting  the  study  described  in  the  preceding  section,  we  observed 
how  subjects  were  sizing  up  a  board  and  understanding  its  dynamics  in  order  to  even  arrive 
at  an  option.  Klein,  Wolf,  Militello,  and  Zsambok  (in  preparation)  have  tried  to  model  the 
factors  that  determine  which  option  a  decision  maker  will  consider. 

Klein  and  Miller  (1992)  have  derived  a  convergence  model  of  how  options  are 
identified.  The  model  consists  of  three  parameters:  the  zone  of  orientation,  contextual 
goals,  and  the  detection  of  leverage  points. 

(1)  Zone  of  orientation.  The  zone  of  orientation  (see  Tikhomirov  &  Poznyanskaya, 
1966)  attracts  attention  because  of  its  potential  information  value—it  is  an  area  of  turbulence, 
of  departures  from  typicality,  of  concentration  of  critical  pieces,  and  so  on.  This  is  the  area 
examined  first. 

(2)  Contextual  goals.  Goals/plans  are  usually  developed  in  response  to  large-scale 
patterns.  Usually,  there  are  general  plans  available  even  before  seeing  the  board  (incr^se 
mobility,  reduce  opponent’s  range),  but  these  are  too  general.  For  a  given  board  position, 
the  player  must  focus  down  to  a  small  set  of  contextually-relevant,  specific  plans,  e.g.,  a 
player  might  notice  that  the  opponent  has  a  Pawn  advantage  at  a  certain  part  of  the  board  and 
may  formulate  the  goal  of  holding  back  these  Pawns.  This  is  followed  by  a  search  for  how 
to  accomplish  this  goal.  De  Groot  (1965,  p.  134)  offers  several  examples:  notice  a  need  to 
do  something  on  the  King’s  wing,  realize  that  the  Rook  must  do  something  since  it  is  the  last 
piece  not  engaged,  consolidate  on  the  Queen’s  side,  embark  on  a  mating  attack,  open  up  the 
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King’s  position,  maintain  pressure.  These  pattern  recognitions  are  followed  by  searches  for 
moves  to  achieve  the  goals. 

(3)  Leverage  points.  Leverage  points  are  departures  from  typicality  that  are  more 
tactical  than  goals/plans.  Holding  (1985,  p.  172)  and  de  Groot  (1965)  refer  to  them  as 
experiential  linkings.  De  Groot  (p.  297)  says  that  "The  extensive  and  subtly  differentiated 
system  of  such  correspondences  (linkings)  is,  so  to  speak,  the  arsenal  of  the 
chessmaster...That  the  chessmaster  sees  in  a  few  s^onds  ‘what’s  cooking  in  a  certain 
position,’  i.e. ,  which  typical  playing  methods  the  situation  on  the  board  demands,  enables 
him  to  begin  his  investigation  in  a  highly  specific  direction." 

These  experiential  linkings  include  patterns  or  Gestalts  such  as:  a  passed  Pawn,  an 
isolated  Pawn,  a  hanging  Bishop,  the  possibility  of  a  series  of  forced  moves.  These  are 
opportunities  as  well  as  threats  to  be  defended  against.  A  chess  player  might  see  that  s/he 
has  a  series  of  forced  attacks  against  the  opponent,  and  muse  "I  wonder  how  I  can  ^e 
advantage  of  that?  If  I  keep  pushing  my  opponent  around,  what  opportunity  might  it  create 

for  me?" 

At  the  simplest,  these  experiential  linkings  are  routine  patterns  such  as  the  fork,  the 
discovered  check,  the  double  check,  the  pin,  the  tempo  move,  Zwischenzug,  Zugzwang,  the 
sacrifice,  and  the  quasi-sacrifice.  De  Groot  (1965)  says  that  the  chessmaster  uses  several  of 
these  patterns,  e.g.,  several  tactical  weaknesses,  to  create  chains  and  multi-functional  moves. 
Further,  these  chains  can  themselves  be  typical  patterns. 

The  story  is  not  so  simple,  since  attempts  to  formulate  simple  rules  run  into 
problems.  "Pawn  grabbing  in  the  opening  is  not  always  fatal;  the  Rook  need  not  always 
stand  behind  the  passed  Pawn;  the  pair  of  Bishops  or  a  Queen  side  Pawn  majority  or  a 
protected  passed  Pawn  or  a  massive  Pawn  center  is  not  always  an  advantage  (de  Groot, 
p.  303). 

At  the  most  complex  level  of  patterns,  are  "all  sorts  of  maneuvers,  offensive  and 
defensive  build  ups,  ways  of  regrouping  the  pieces  and  getting  them  to  cooperate,  etc.  The 
general  methods  of  piece  co-operation  appear  to  be  particularly  difficult  to  describe  in  detail, 
to  treat  systematically,  and  therefore  to  pass  on  to  other  students  of  the  game  (de  Groot, 
p.  303). 

These  leverage  points  are  not  easily  definable,  and  according  to  de  Groot,  players  at 
different  strengths  will  show  different  speeds  of  situation  assessment  for  detecting  these 
critical  contextual  features.  A  strong  player  will  immediately  notice  the  threat  of  a  pin,  and 
realize  the  need  to  drive  the  opponent’s  Queen  away  and  remove  the  threat.  De  Groot 
argues  that  these  are  not  universals,  but  just  features  that  are  striking  in  context,  as 
deviations  from  typicality.  In  other  words,  they  are  defining  features  of  a  position,  as  well 
as  clarion  calls  for  reacting:  How  to  take  advantage  of  that  passed  Pawn?  How  to  overcome 
my  opponent’s  space  advantage?  How  to  use  my  pair  of  Bishops? 
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So,  these  leverage  points  or  experiential  linkings  may  be  describable  at  the  level  of 
novice,  but  not  at  the  level  of  Master.  That  is  a  reason  to  restrict  our  investigation  to 
novices. 

The  Convergence  model  of  option  generation  asserts  that  option  generation  occurs  as 
a  top-down  process  of  planning,  together  with  a  bottom-up  process  of  noticing  situational 
opportunities.  Where  these  correspond,  an  option  arises. 

The  metaphor  is  a  lightning  strike,  which  is  the  convergence  of  two  streamers,  one 
positive  and  one  negative.  For  a  lightning  strike  to  occur,  first  there  needs  to  be  a  cloud, 
probably  at  a  suitable  temperature  (cool)  to  allow  the  positive  and  negative  charges  inside  Ae 
cloud  to  become  separated  so  that  the  negative  charges  are  at  the  bottom.  ^^Tien  the  negative 
charge  is  great  enough  to  overcome  the  air’s  resistance,  a  streamer  of  flowing  electrons, 
called  a  stepped  leader,  is  discharged  towards  the  ground.  At  the  same  time,  the  negative 
charge  attracts  a  weak  positive  charge  upward  from  the  earth’s  surface,  a  positive  streamer. 
As  the  stepped  leader  continues  downward,  it  continues  to  attract  the  positive  streamer 
upward.  When  these  meet,  the  circuit  is  completed,  and  the  lightning  bolt  is  discharged. 

For  option  generation,  the  zone  of  orientation  is  analogous  to  the  cloud.  This  is  the 
arena  in  which  the  option  will  become  generated.  The  separation  of  positive  and  negative 
charges  in  the  cloud  is  akin  to  the  culling  out  of  contextually  relevant  plans  from  the  general 
set  of  possible  plans.  These  specific  plans  help  to  focus  the  option  generation  process,  and 
help  to  attract  the  right  option.  The  specific  plans  are  like  stepped  leaders,  opening  up 
promising  channels  for  investigation.  The  leverage  points,  or  experiential  linkings,  are  like 
the  positive  streamers  lifting  off  of  the  ground.  They  take  advantage  of  fortuitous  features  of 
geography,  such  as  trees  or  buildings.  In  a  chess  game,  atypical  features  such  as  a  passed 
Pawn  or  an  undefended  Bishop  naturally  trigger  curiosity  about  how  to  take  advantage  of  a 
flaw,  or  how  to  defend  against  a  flaw.  So  the  person  is  simultaneously  alert  to  possibilities 
for  using  a  feature  of  the  game  and  for  ways  to  modify  the  game  position  in  certain 
directions.  As  these  two  lines  of  investigation  grow  nearer,  e.g.,  deepening  from  a  leverage 
point  starts  to  get  in  contact  with  a  plan,  this  line  of  investigation  excites  more  attention  and 
if  it  is  possible  to  make  the  contact,  the  option  is  generated. 

Sometimes,  the  process  will  primarily  be  bottom  up,  building  from  a  promising 
leverage  point.  Other  times,  the  process  may  be  primarily  top-down,  searching  for  ways  to 
carry  out  a  plan.  A  third  pattern  is  a  fairly  equivalent  direction  from  above  and  from  below. 

The  Convergence  model  of  option  generation  seems  to  be  relevant  to  areas  other  than 
chess.  Smith  et  al.  (1991)  have  described  how  specialists  perform  blood  typing  in  blood 
bank  centers,  using  top-down  and  bottom  up  processes  that  correspond  fairly  well  to  the 
processes  in  the  Convergence  model.  Military  planning  also  calls  for  option  generation. 

With  experience,  commanders  are  able  to  rapidly  focus  on  a  critical  part  of  the  battlefield 
map  (the  zone  of  orientation),  and  are  able  to  both  search  for  ways  to  carry  out  plans  and  to 
notice  leverage  points  in  the  battlefield  to  drive  the  situation  assessment. 
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Therefore,  we  suggest  that  the  Convergence  model  of  option  generation  may  have 
value  for  understanding  and  assessing  the  processes  found  in  operational  settings  as  well  as 
more  controlled  tasks. 


To  summarize,  there  are  several  aspects  of  the  Convergence  model  of  option 
generation. 

_ consists  of  three  components,  zone  of  orientation,  contextual  goals,  and  leverage 

points. 

-  Each  of  these  components  can  be  further  specified  for  a  given  board  position,  and 
for  a  given  player.  That  is,  players  have  certain  types  of  plans  and  certain  types  of  Gestalts, 
they  notice  more  quickly. 

-  The  zone  of  orientation  defines  where  the  option  will  arise. 

--  The  contextual  goals  and  triggers  are  the  bases  for  the  option.  The  plan  is  the 
top-down  portion,  and  the  leverage  points  are  the  bottom-up  portion.  The  plan  is  looking  for 
a  way  to  be  realized,  and  the  leverage  points  are  looking  for  a  function  to  serve. 

-  The  Convergence  model  is  not  a  radical  departure.  It  is  consistent  widi  the  work 
of  Chase  and  Simon  (1973),  de  Groot  (1965),  and  Holding  (1985).  Hopefully,  it  is  a  little 
more  explicit  and  therefore,  perhaps  more  useable.  Chase  and  Simon  talked  about  pattern 
matching  to  generate  moves,  whereas  the  Streamer  model  relies  on  pattern  matching  for 
experiential  linkings  (triggers),  which  helps  start  the  process  of  option  generation  but  leaves 
room  for  the  chess  player  to  construct  the  move  sequence,  rather  than  just  recognizing  it. 

This  was  also  de  Groot’s  concern,  and  Holding’s— that  option  generation  could  not  be  a 
simple  retrieval  process.  The  Convergence  model  reflects  the  importance  of  low-level 
patterns  (e.g.,  a  passed  Pawn),  while  maintaining  the  potential  for  directed  search  and 

discovery. 

-  The  Convergence  model  is  consistent  with  earlier  work  on  problem  solving 
(Duncker,  1945;  Klein  &  Weitzenfeld,  1978).  This  work  viewed  problem  solving  as 
consisting  of  two  interrelated  processes;  identifying  the  goal  and  searching  for  solutions. 
lU-defmed  problems  need  to  do  both  simultaneously.  The  Convergence  model  also  views  a 
relationship  between  specifying  the  plans  and  worldng  up  from  leverage  points.  The 
promising  triggers  are  searching  for  reasonable  plans,  and  the  plans  are  searching  for  ways  to 
be  accomplished,  so  that  plan  specification  is  continuing  even  as  the  experiential  linkinp  are 
being  explored.  That  is  why  a  purely  top-down  approach  doesn  t  work  (i.e.,  first  sp^ify 
plans,  then  look  for  ways  to  accomplish  them),  and  why  a  bottom-up  approach  doesn’t  work 
(i.e.,  mindlessly  explore  combinations). 

T, imitations  of  Recoenitional  Decision  Making 


In  this  section  we  examine  some  of  the  weaknesses  of  using  the  RPD  strategy.  Klein 
(1989)  has  noted  that  classical  strategies  of  concurrent  option  evaluation  are  more  suitable  for 
tasks  where  there  are  abstract  data,  tasks  requiring  optimizing  rather  than  satisficing,  tasks 
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requiring  the  choice  to  be  justified,  tasks  requiring  agreement  from  multiple  stakeholders, 
combinatorial  tasks,  and  tasks  where  the  level  of  expertise  is  low.  It  is  therefore  apparent 
that  we  do  not  view  the  RPD  strategy  as  appropriate  for  all  tasks,  or  for  all  people.  In  this 
section  we  further  explore  some  of  the  boundary  conditions  of  the  RPD  strategy.  We  begin 
by  reviewing  these  task  characteristics  just  mentioned.  Then,  we  describe  a  study  of  the 
types  of  errors  found  in  our  naturalistic  decision  studies.  Finally,  we  examine  the  impact  of 
stress  on  recognitional  decision  making. 

Generally,  the  RPD  model  will  have  a  greater  advantage  over  classical  models  when 
complex  stimuli  are  involved,  rather  than  alphanumeric  data.  However,  the  RPD  model  is 
still  applicable  with  data  such  as  symbols  on  a  CRT,  or  even  numbers  in  a  table.  But  as  the 
data  and  format  become  more  abstract,  the  advantages  of  the  strategy  diminish.  Similarly, 
the  recognitional  strategy  is  designed  for  tasks  where  people  try  to  satisfice,  rather  than 
optimize,  but  even  for  an  optimizing  task  such  as  chess,  de  Groot  has  presented  protocols 
showing  serial  evaluation  rather  than  comparisons  between  options.  With  regard  to 
experience  level,  we  have  found  a  lower  rate  of  use  of  recognitional  strategies  among  novice 
decision  makers,  but  the  rate  is  still  quite  high,  never  less  than  35%  in  any  of  die  studies 
conducted  thus  far.  The  other  criteria  seem  to  be  firmer:  requirement  for  justification, 
agreement  from  multiple  stakeholders,  and  combinatorial  tasks  all  mitigate  against  the  use  of 
the  RPD  strategy. 

What  kinds  of  errors  arise  from  the  use  of  a  recognitional  strategy?  Klein  (in 
preparation)  investigated  25  errors,  identified  from  a  careful  review  of  a  series  of  studies 
summarized  in  Klein  (1989)  and  other  sources,  and  covering  more  than  450  decision  points 
that  had  been  thoroughly  analyzed.  The  errors  fell  into  three  categories.  The  first  class, 
containing  16  errors,  was  attributed  to  a  lack  of  experience,  which  typic^y  resulted  in 
failure  to  anticipate  certain  consequences  of  actions  or  in  misunderstanding  of  analogues. 

The  second  class,  containing  five  errors,  was  made  up  of  incidents  in  which  a  lack  of  critical 
information  led  to  the  problem.  The  third  class,  containing  four  errors,  was  linked  to  faulty 
mental  simulation,  e.g.,  cases  where  a  critical  sign  of  illness  in  a  patient  was  explained  away 
by  constructing  a  story  that  showed  how  that  sign  could  be  the  byproduct  of  some  other 
factor. 


It  is  noteworthy  that  in  these  25  cases,  there  was  not  a  single  example  of  errors  due  to 
miscalculation  of  probabilities,  poor  weighing  of  evaluation  dimensions,  mistaken  assignment 
of  subjective  expected  utilities,  confirmation  bias,  etc.  The  implication  of  these  error  data  is 
that  RPD  errors  arise  from  limited  experience  that  renders  a  person  insensitive  to  factors  that 
should  have  influenced  a  situation  assessment,  and  from  ineffective  use  of  mental  simulation 
for  explaining  a  situation  or  for  evaluating  a  course  of  action.  Klein  and  Crand^l  (in 
preparation)  have  argued  that  mental  simulation  can  lead  to  several  types  of  decision  errors, 
including  overconfidence  in  the  likelihood  that  a  course  of  action  will  be  successful, 
explaining  away  disconfirming  evidence  by  finding  counterexplanations,  and  failing  to 
perform  a  thorough  mental  simulation  to  investigate  the  risks  of  carrying  out  a  course  of 
action. 


Another  way  that  the  quality  of  decisions  can  be  degraded  is  through  the  effect  of 
stress.  Recently,  Driskell  and  Salas  (1991)  have  presented  an  exhaustive  review  of  the 


27 


literature  on  different  types  of  stressors.  Klein  (1991;  pein  &  Zsambok,  in  preparation)  has 
attempted  to  apply  this  review  to  the  topic  of  naturalistic  decision  making.  Klein  and 
Zsambok  (1991)  have  argued  that  the  diverse  types  of  stressors,  e.g.,  cold,  noise,  sleep 
deprivation,  time  constraints,  frustration,  performance  pressure,  etc.,  share  a  few  features  in 
common.  First,  they  affect  the  speed/accuracy  tradeoff,  and  their  effect  on  decision  m^ng 
can  largely  be  explained  in  terms  of  less  time  being  allotted  to  sub-tasks  such  as  gathering 
information.  Second,  each  stressor  requires  attention  for  adjusting  to  and  managing  the 
stress  itself,  and  this  may  be  seen  as  constituting  a  secondary  task,  so  that  degraded 
performance  under  stress  may  be  also  a  function  of  fewer  cognitive  resources  available. 

Within  the  RPD  model,  the  most  vulnerable  as^t  is  mental  simulation.  R^gnizing 
a  situation  does  not  require  much  in  the  way  of  cognitive  effort,  but  niental  simulation 
requires  careful  attention,  and  is  easUy  disrupted.  Therefore,  we  predict  that  stress  will  most 
easUy  degrade  the  mental  simulation  function  and  disrupt  recognitional  decision  making 
through  that  avenue. 


Team  Decision  Makine 


Can  we  apply  our  ideas  about  individual  decision  making  to  the  level  of  the  team? 

This  research  project  has  centered  around  individual  decision  making,  as  captured  by  the 
RPD  model,  but  we  have  also  become  interested  in  teams.  When  we  studied  urban 
fireground  commanders,  we  watched  how  they  managed  their  teams  during  time  pressured 
events.  When  we  studied  Incident  Commanders  in  charge  of  fighting  forest  fires,  we  were 
interested  in  the  way  they  managed  very  large  teams  and  even  an  entire  organization  of  4,000 

people. 

Team  as  an  Entity 

Our  first  attempt  to  describe  team  decision  making  was  to  export  the  RPD  model 
wholesale  to  the  team  level,  and  to  treat  the  team  as  an  entity.  Thordsen  and  Klein  (1989) 
hypothesized  that  the  RPD  model  could  be  translated  to  the  team  level  by  corresponding  the 
primary  components  of  the  RPD  model-recognizing  the  situation,  identifying  goals, 
becoming  sensitive  to  critical  cues,  forming  expectancies,  identifying  a  feasible  course  of 
action,  conducting  a  mental  simulation  to  evaluate  that  course  of  action— to  the  following 
correspondences  at  the  team  level.  Recognizing  a  situation  for  a  team  is  linked  to  making 
sure  that  there  is  a  shared  understanding  (what  Orasanu  &  Salas,  in  press,  have  referr^  to  as 
a  shared  mental  model).  Identifying  goals  corresponds  to  communicating  the  leader’s  intent. 
Becoming  sensitive  to  critical  cues  corresponds  to  effective  communication  about  information 
signifying  changes  in  the  environment.  Forming  expectancies  corresponds  to  sharing  an 
understanding  between  team  members  about  expectancies.  Identifying  a  feasible  course  of 
action  corresponds  to  sharing  an  understanding  of  how  to  approach  the  problem.  Mental 
simulation  is  also  found  at  the  team  level,  as  an  activity  in  which  the  entire  team  can 
participate,  a  phenomenon  we  observed  during  an  ARTBASS  simulation  at  Ft.  Hood 
(Thordsen,  Galushka,  Klein,  Young,  &  Brezovic,  1990). 

Thordsen  (in  preparation)  has  tested  this  approach  by  using  these  categories  to 
generate  a  taxonomy  of  team  errors.  The  different  taxonomy  elements  corresponded  to 
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different  RPD  functions,  with  one  additional  function  for  management  of  the  team.  He 
hypothesized  that  if  this  taxonomy  was  reasonable  and  intuitive,  then  different  rates  should 
reliably  generate  overlapping  ratings. 

Thordsen  developed  as  stimulus  materials  65  team  errors  that  Klein  Associates  had 
observed  in  four  different  domains.  These  included  24  errors  from  a  Department  of  Energy 
emergency  operations  center  exercise,  21  errors  from  tactical  military  exercises,  14  errors 
from  full  flight  simulations  for  commercial  airline  crews,  and  six  errors  from  a  strategic 
geopolitical  exercise  at  the  National  Defense  University.  Three  coders  were  provided 
packages  containing  a  set  of  eight  practice  errors,  descriptions  of  the  65  team  errors,  and  a 
detailed  outline  of  the  error  taxonomy  and  its  relationship  to  the  team  decision  model. 

Inter-coder  agreement  was  assessed  using  the  Kappa  coefficient  (Fleiss,  1981).  Kappa 
is  a  chance-related  measure  of  agreement  that  provides  a  more  meaningful  index  of 
inter-rater  reliability  than  do  simple  percent-of-agreement  measures.  Kappa  values  for  the 
overall  level  of  category  agreement  between  pairs  of  coders  were  .69,  .70,  and  .74.  Kappa 
values  greater  than  .75  represent  excellent  agreement,  and  values  between  .40  and  .75 
indicate  fair  to  good  agreement. 

We  were  very  interested  in  the  issues  of  what  differentiates  domains  from  each  other 
and  whether  it  is  possible  to  gain  insight  into  relationships  between  a  domain’s  attributes  and 
the  categories  into  which  its  errors  fall.  Eight  attributes  were  identified,  based  on  our 
experiences  observing  teams  in  naturalistic  settings,  along  which  we  believed  the  domains 
would  vary.  These  were  the  level  of  Uncertainty,  the  amount  of  Time  Pressure,  the  level  of 
Responsibility  and  Risk  of  and  to  the  team  members,  the  average  level  of  individual 
Expertise,  the  amount  of  team  Cohesion,  the  Dynamics  of  the  environment  they  were 
working  within,  the  Complexity  of  the  tasks  they  must  perform,  and  the  degree  of 
Distribution  of  the  team  members.  While  no  list  would  be  exhaustive,  we  believe  this  one 
captures  many  of  the  critical  factors. 

The  data  revealed  some  indications  of  relationships  between  domain  attributes  and 
error  type.  For  example,  there  were  no  Cues  errors  from  the  domain  with  the  lowest 
Dynamic  rating.  In  a  non-dynamic  environment,  the  Cues  that  are  present  often  remain^ 
constant  and  few  new  Cues  are  introduced.  The  vast  majority  of  Goal  errors  were  attributed 
to  the  domain  which  possessed  high  Uncertainty  and  a  high  Dynamic  environment.  Both  of 
these  characteristics  might  make  it  difficult  to  develop  adequate  goals,  especially  under  high 
time  pressure,  which  this  domain  also  possessed.  This  same  domain  also  had  the  largest 
number  of  Evaluation  errors.  Once  again,  the  high  uncertainty,  dynamics,  and  time  pressure 
could  easily  account  for  this  trend  since  it  would  be  difficult  to  quickly  evaluate  unclear, 
changing  situations.  Implementation  errors  were  largely  dominated  by  the  only  domain  in 
which  the  team  members  were  directly  responsible  for  the  actual  implementation  of  courses 
of  action.  The  other  domains  operated  more  in  planning  and  managing  capacities.  One  team 
dominated  the  Management  errors.  This  particular  team  was  comprised  of  individuals  who 
seldom  worked  together  and  whose  areas  of  expertise  did  not  match  the  work  that  their  team 
was  expected  to  accomplish,  that  is,  they  had  the  least  team  Cohesion  and  the  lowest 
individual  Expertise. 
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The  study  of  team  errors  demonstrated  that  it  is  possible  to  consider  a  team  as  an 
intelligent  entity,  and  to  apply  the  perspective  of  an  individual  decision  maker  to  a  team.  We 
originally  conceived  the  study  as  a  test  of  the  RPD  model,  and  developed  it  using  the  RPD 
model  as  the  conceptual  frame.  But  over  the  course  of  this  study  and  other  ongoing  work 
devoted  to  team  decision  making,  we  have  come  to  believe  that  the  RPD  model  is  too 
narrowly  focused  on  particular  aspects  of  decision  making  and  does  not  adequately  reflect  the 
breadth  or  variety  of  team  decision  processes.  Rather  than  trying  to  force  our  analyses  of 
teams  into  the  RPD  model,  our  more  recent  work  has  used  a  more  general  cognitive  process 
model  of  team  decision  maldng.  The  Team  Error  study  was  a  valuable  first  step  in 
development  of  a  cognitive  perspective  on  team  functioning. 

Cognitive  Model  of  Team  Development 

In  this  section  we  describe  a  cognitive  developmental  model  of  team  decision  making. 
The  model  treats  teams  as  intelligent  entities,  and  uses  our  knowledge  of  cognitive 
psychology  and  developmental  psychology  as  a  source  of  hypotheses  and  guidance.  The  use 
of  metaphors  in  science  (e.g.,  Hesse,  1966)  has  been  estabUshed  as  a  means  of  generaung 
new  hypotheses.  Thus,  a  cognitive  metaphor  alerts  us  to  the  needs  a  team  has  to  direct 
attention,  to  store  information,  to  retrieve  information  from  a  distributed  memory ,  to  develop 
metacognitive  skills,  and  so  forth.  In  this  way,  cognitive  psychology  provides  a  perspective 
for  understanding  teams,  and  for  identifying  processes  in  which  teams  must  engage. 

Similarly,  developmental  models  serve  as  a  metaphor  for  understanding  a  maturing  organism. 
Functions  such  as  the  formulation  of  identity  and  the  achievement  of  cognitive  sophistication, 
as  well  as  the  development  of  metacognitive  skills,  all  draw  on  the  research  literature  on  the 
cognitive  development  of  children. 

There  is  another  way  of  viewing  a  cognitive  development  metaphor,  and  that  is  to  use 
teams  to  understand  individuals®.  Max  Black  (1962)  has  asserted  that  metaphors  are 
interactive.  To  say  that  man  is  a  wolf  changes  the  way  we  perceive  men,  and  also  the  way 
we  perceive  wolves.  To  treat  a  team  as  an  intelligent  entity  requiring  copitive  development 
can  aid  us  in  understanding  teams,  but  it  can  also  inform  us  about  cognitive  development. 
Specifically,  in  our  research  on  teams  we  have  seen  how  important  and  difficult  the  tok  of 
forming  and  communicating  intent  is,  and  this  raises  questions  about  the  function  of  intent  m 
directing  individual  cognition.  In  our  work  with  teams,  we  have  observed  how  difficult  it  is 
to  draw  on  the  different  team  members  to  formulate  ideas.  No  one  in  the  team  knows  what 
the  final  ideas  will  be  during  a  planning  session.  Different  members  contribute  different 
components,  and  each  member  may  have  an  idiosyncratic  interpretation  of  the  final  product. 
This  chaotic  process  may  map  onto  individual  cognition,  where  different  experiences 
somehow  converge  into  an  idea,  a  product  that  was  not  previously  known.  Therefore,  the 
study  of  thinking  in  teams  may  help  provide  hypotheses  about  the  process  of  individual 
cognition. 

The  cognitive  model  of  team  development  was  first  articulated  by  Thordsen  and  Klein 
(1989),  who  specifically  used  cognitive  processes  such  as  perception,  metacognition. 


®This  suggestion  was  made  by  Gavem  Lintem,  personal  communication,  April,  1991. 
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intention,  and  limited  attention  to  describe  the  cognitive  behavior  of  teams.’  The  cogmtive 
model  was  then  applied  in  a  study  of  teamwork  among  commercial  aviation  cockpit  crew 
members  (Klein  &  Thordsen,  1989).*  The  cognitive  model  was  further  elaborated  to  include 
the  process  of  cognitive  development  as  a  way  of  describing  the  growth  in  teamwork  skills, 
this  was  accomplished  during  a  study  of  battalion  teams  functioning  in  field  conditions 
(Thordsen,  Klein,  &  Wolf,  1991).’  Finally,  under  the  auspices  of  the  present  basic  research 
contract,  we  have  extended  the  developmental  model  even  further,  as  described  below. 

Thus,  we  have  come  full  cycle  during  this  three-year  contract,  from  the  initial  formulation  of 
a  model  of  teamwork,  to  a  much  more  detailed  account  based  on  several  field  studies. 

The  Advanced  Decision  Making  Model  (Zsambok,  Klein,  Kyne,  &  Klinger, 
1992)  describes  how  teams  become  a  collective  body  that  is  capable  of  high  performance  and 
advanced  decision  making.  The  model  is  stated  in  developmental  terms.  It  includes  two  key 
characteristics  of  decision  making  teams-zetzm  identity  and  conceptual  level-zad  one  master 
regulating  process-self  monitoring.  As  teams  develop  from  a  stote  of  weak  to  strong  team 
identity,  from  a  low  to  high  conceptual  level,  and  from  lax  to  vigilant  self  monitoring,  they 
become  advanced  decision-making  teams. 


The  model  also  describes  processes  by  which  teams  may  improve  in  each  of  the  key 
components  of  team  decision  making.  Figure  9  depicts  the  critical  processes  that  are  related 
to  each  of  the  components.  As  teams  sharpen  their  ability  to  use  these  proces^s,  they 
strengthen  their  team  identity,  raise  their  conceptual  level,  and  improve  in  their  overall 
ability  to  self  monitor.  The  processes  are  ones  that  observers  can  learn  to  see  in  teams-with 
practice,  team  members  can  learn  to  see  them  too.  Through  observation,  feedback,  and 
practice  with  these  processes  that  groups  of  individuals  can  develop  to  advanced,  high 
performance  decision-making  teams. 

The  three  components  of  teams  are  parallel  to  dimensions  frequently  used  to  observe 
an  individual  who  is  trying  to  develop  a  skill.  Inexperienced  teams  go  through  ^1  the  ^ 
awkwardness  and  confusion  of  novices.  For  example,  novices  have  difficulty  with  their  own 
identity,  so  they  often  can’t  judge  which  tasks  they  will  be  able  to  successfully  perform. 
Novices  have  difficulty  in  pursuing  long  range  goals  without  getting  distract^.  Novices  c^ 
be  comical— watch  someone  step  onto  a  racquetball  court  for  the  first  time  with  no  idea  which 
way  the  ball  will  ricochet.  Novices  also  get  confused  when  they  have  to  view  a  problem 
from  more  than  one  perspective,  and  they  are  unable  to  think  about  their  own  thinking— to 
gauge  what  strategy  to  use  in  performing  a  task. 


’This  article  followed  the  work  of  Thordsen  in  formulating  his  study  design,  but  it  was 
completed  earlier  since  the  Thordsen  project  involved  running  subjects  and  analyzing  data. 

*This  project  was  supported  by  a  contract  from  NASA/ Ames. 

mUs  study  was  sponsored  by  the  ARI  field  unit  at  Monterey,  through  SBIR  Contract 
No.  MDA903-90-C-0119. 
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Figure  9.  Key  behaviors  for  advanced  team  decision  making. _ 

This  model  was  developed  to  help  teams  learn  to  identify  specific  processes  they  can 
use  to  improve  their  performance— to  help  them  develop  into  an  advanced  decision-making 
team.  A  brief  description  of  each  of  the  processes  follows: 

Self  Monitoring 

An  advanced  team  monitors  itself  while  acting  within  the  task,  and  balances  the  ne^ 
for  time  to  monitor  with  that  required  to  perform  the  task.  By  definition,  self  monitoring  is 
the  process  of  team  members  observing  and  assessing  their  own  team  performance.  But  the 
term  also  implies  that  they  try  to  adjust  various  aspects  of  performance  that  have  been  found 
wanting.  In  advanced  teams,  adjusting  occurs  interactively-watching,  adjusting,  watching 
again,  and  so  forth.  One  critical  aspect  of  self  monitoring  concerns  time  management-hemg 
able  to  meet  goals  at  the  desired  level  of  quality  before  deadlines  overtake  the  team. 

Self  monitoring  is  a  regulatory  process  that  encompasses  all  other,  more  specific 
processes  related  to  team  identity  and  cognitive  level.  Each  of  these  is  discussed  below. 

Just  as  teams  can  develop  and  improve  in  their  use  of  these  specific  processes,  they  can  also 
improve  in  their  ability  to  engage  in  the  process  of  monitoring  itself. 
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Team  Identity 


Team  identity  reflects  the  extent  to  which  members  conceive  of  themselves  as  an 
interdependent  unit,  and  operate  from  that  perspective  while  engaged  in  their  tasks.  One  key 
outcome  of  strong  team  identity  is  that  members  can  anticipate  the  needs  and  actions  of  one 
another  thereby  facilitating  coordination  of  thoughts  and  actions  across  the  team.  Te^s  can 
advance  in  the  strength  of  their  identity  by  developing  the  way  they  use  certam  critical 
processes; 


ne.fininp  roles  and  functions.  Team  members  cannot  monitor  whether  the  functions 
assigned  to  particular  roles  are  being  properly  carried  out  if  they  don’t  toow  what  those  roles 
and  functions  are.  Advanced  teams  recognize  that  they  must  understand  the  pattern  of 
working  relationships  among  their  members  in  order  for  them  to  coordinate  activiUes  across 

the  team. 


Engaging.  The  extent  to  which  members  are  fully  absorbed  in  the  roles  and  functions 
that  need  to  be  performed  to  achieve  team  goals  is  an  index  of  team  engagement.  More 
experienced  teams  are  careful  to  get  the  fuU  use  of  each  member,  so  they  monitor  themselves 
to  see  if  any  members  have  become  disengaged  from  the  task  at  hand. 

Compensating.  Compensating  describes  the  process  of  stepping  out  of  an  ^signed 
role  or  function  and  taking  on  a  different  one  to  help  the  team  reach  its  goals.  Advanced 
teams  encourage  members  to  flexibly  step  outside  their  roles  as  the  need  anses  m  order  to 
remedy  difficulties  that  other  members  are  having  in  fulfilling  their  funcbons. 

Avoiding  micromanagement.  Micromanagement  occurs  when  a  team  member 
becomes  involved  in  managing  information,  tasks,  or  people  at  a  greater  level  of  detail  than 
is  necessary.  Advanced  teams  realize  that  micromanagement  drmns  the  team  of  resources  in 
two  ways:  1)  the  team  member  engaged  in  micromanaging  is  distracted  from  other,  hig  er 
level  tasks;  and  2)  the  micromanager  often  dupUcates  what  others  are  doing,  or  oAerwise 
interferes  with  another  member’s  work.  As  teams  develop  in  their  ability  to  avoid 
micromanagement,  they  recognize  the  need  to  monitor  for  this  problem,  and  to  resolve  it 

quickly  when  it  does  occur. 


Conceptual  Level 

Advanced  teams  are  able  to  achieve  a  more  complex  and  sojphisticated  understanding 
of  tasks,  events,  and  situations.  They  are  able  to  employ  hypothetical  reasoning,  carry  out 
contingency  planning,  and  manage  divergent  ideas.  In  contrast,  less  advanced  terns 
crystalize  their  thinking  around  a  few  concrete  courses  of  action  even  when  the  situation 
requires  more  flexibility.  A  key  outcome  of  developing  a  high  conceptual  level  is  that  team 
members  are  able  to  improvise  quickly  and  effectively  in  the  face  of  unexpected  events. 
Processes  critical  to  a  teams’  conceptual  level  include: 

Envisioning  goals  and  plans.  The  process  of  envisioning  involves  bringing  to  Ufe  in 
realistic  and  graphic  terms  the  mission  (goals)  of  the  team,  and  the  process  (plan)  the  team 
will  use  to  meet  its  goals. 


33 


FnriK-  Time  horizon.  The  Time  Horizon  is  the  focal  distance  at  which  a  team  is 
perceiving  the  world.  Whether  a  team  is  using  an  appropriate  Time  Horizon  is  a  direct 
function  of  the  task  in  which  they  are  engaged.  For  example,  for  a  helicopter  crew,  the 
Time  Horizon  may  be  the  map  cue  that  is  just  beyond  the  next  visible  navigation  m^ker. 

For  a  Division  planning  team,  the  appropriate  Time  Horizon  might  be  24-72  hours  m  the 
future.  For  high  level  planning  teams,  the  appropriate  Time  Horizon  might  be  5-10  years 

into  the  future. 

Focus:  Range  of  factors  considered.  Another  aspect  of  perception  that  changes  as  a 
team  develops  is  the  range  of  factors  monitored.  The  team  becomes  sensitive  to  a  wider  set 
of  causal  factors,  and  learns  to  apportion  its  attentional  resources  so  that  different  members 
can  capture  different  types  of  information. 

Detecting  gaps  and  ambiguity.  Advanced  teams  actively  attempt  to  detect  gaps  in 
information  by  scrutinizing  what  they’ve  been  given,  and  by  clarifying  their  assumptions 

about  the  information  base.  Advanced  teams  are  also  not  perplexed  by  ambiguous 

information.  They  make  note  of  the  ambiguity,  rather  than  ignonng  it  or  becoming  confused 
by  it. 

Converging  on  situation  assessment.  Advanced  teams  work  towards  developing  a 
shared  understanding  across  team  members  of  a  commonly  held  situation  assessment.  Under 
rapidly  changing  conditions,  advanced  teams  recognize  the  need  to  re-assess  the  situation  and 
to  ensure  that  shifts  in  SA  are  understood  by  all  members. 

^pftkinp  divergence.  Advanced  teams  explicitly  seek  divergent  views  from  their 
members  to  sharpen  and  deepen  their  situation  assessments  and  plans  of  action.  They 
encourage  diversity  rather  than  suppressing  it  as  an  unwanted  complication. 


The  Importance  of  Teamwork  and  Intent 

We  have  conducted  a  study  of  one  of  these  components,  that  aspect  of  a  team  s 
identity  by  which  it  learns  how  to  communicate  about  its  own  goals.  This  study  (Kaempf, 
Klein,  &  Kyne,  in  preparation)  examined  goal  communication  under  field  conditions.  We 
tried  to  understand  the  elements  that  contribute  to  the  effectiveness  of  commander’s  intent 
statements.  There  were  several  objectives  for  this  analysis.  The  first  was  to  identify  the 
characteristics  of  effective  commander’s  intent  statements.  The  second  objective  was  to 
determine  if  these  characteristics  depend  on  the  type  of  mission.  The  final  obj^tive  w^  to 
examine  if  the  commander  learns  to  convey  better  intent  statements  across  missions  withm 

the  training  rotation. 

The  intent  statements  were  obtained  from  mission  orders  created  during  force  on  force 
exercises  at  the  National  Training  Center  (NTC).  Individual  commanders  participate  in  a 
series  of  missions  called  a  rotation.  Each  rotation  is  made  up  of  one  mission  per  day  or 
approximately  two  weeks.  Because  of  the  nature  of  battle,  the  comm^der  executes  different 
types  of  missions.  For  example,  his  mission  might  be  to  attack,  to  initiate  a  movement  to 
contact,  or  to  build  a  defense  in  a  sector.  The  type  of  mission  executed,  of  course,  depends 
on  the  play  of  the  battlefield. 
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We  received  mission  orders  from  11  rotations  in  successive  order,  just  as  they  had 
been  played  out  at  NTC.  Each  rotation  contained  approximately  seven  mtent  statements 
(range;  five  to  nine).  The  analysis  of  this  data  is  a  non-intrusive  means  of  studymg  the  way 
intent  statements  are  framed,  and  how  they  change  with  comm^d  experience.  The 
commander’s  intent  statements  were  extracted  from  actual  nussion  orders. 

Questionnaires  were  distributed  to  17  volunteers  from  three  operational  settings.  Of 
those  seven  completed  questionnaires  were  returned.  These  questionnaires  solicited  opimons 
about  the  quality  and  content  of  35  statements  of  commander’s  intent  extracted  from  mission 
orders  created  during  force  on  force  exercises  at  the  National  Training  Center.  Those 
settings  include; 

Armv  War  College.  Questionnaires  were  distributed  to  ten  volunteers  enrolled  in 
courses  at  the  Army  War  CoUege  (AWC),  Carlisle  Barracks,  PA.  One  volunteer  returned  a 
completed  questionnaire. 

Former  National  Training  Center  commanders.  We  distributed  questionnaires  to  two 
Army  personnel  who  have  had  extensive  experience  as  commanders  at  the  NTC.  One  se^ed 
as  a  commander  for  the  Red  Forces,  the  other,  who  is  now  a  brigadier  general,  conducted 
field  exercises  and  rotated  forces  through  NTC.  Both  people  returned  a  completed 
questionnaire. 

Instructors  from  the  Tactical  Commander  Development  Course  at  Fort  Leavenworth, 
Kansas.  Questionnaires  were  distributed  to  five  retired  Army  personnel  and  instructors  at 
Fort  Leavenworth.  Four  returned  completed  questionnaires. 

The  questionnaire  consisted  of  three  parts.  The  first  part  contained  questions  that 
represent  the  participants’  level  of  experience.  The  second  part  listed  the  elements  we  have 
identified  as  contributing  to  effectiveness  of  commander’s  intent  statements.  The  evaluator 
was  asked  to  rank  those  elements  in  terms  of  their  importance  for  conveying  intent 
effectively.  The  evaluator  was  also  asked  to  include  any  elements  that  were  missing  from 
our  list.  They  were  then  asked  to  reorder  them  in  terms  of  importance  to  include  their 
elements.  The  third  part  of  the  questionnaire  was  the  35  intent  statements  extracted  from  the 
77  mission  orders  created  at  the  National  Training  Center. 

Evaluator  expertise.  The  level  of  experience  and  rank  reported  by  the  evaluators 
indicated  a  highly  experienced  group.  The  average  length  of  active  duty  was  24  years,  the 
minimum  being  21  years,  and  the  maximum  29  years.  One  of  the  evaluators  was  a  Bngadier 
General,  two  were  Colonels,  and  four  were  Lieutenant  Colonels.  Having  battalion  and/or 
brigade  level  command  experience  was  an  important  basis  for  selecting  these  evaluators.  All 
evaluators  reported  having  battalion  level  command  experience,  the  average  bemg  2.4  y^s 
with  a  range  of  1.5  to  4  years.  One  evaluator  also  reported  having  23  months  of  Brigade 

command  experience. 

Experience  as  an  operations  officer  was  not  a  criterion  used  to  select  our  evaluators 
for  this  study,  however  we  hoped  that  some  would  enjoy  such  experience.  As  it  turns  out. 
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five  of  the  seven  evaluators  reported  having  over  one  year  of  battalion  and/or  brigade  level 
experience  as  an  operations  officer. 

From  these  reports  of  experience  level  and  from  the  fact  that  we  selected  our 
evaluators  from  areas  in  the  Armed  Services  where  proficiency  with  operation  orders  is  high, 
we  are  confident  that  our  data  were  coUected  from  evaluators  who  are  highly  qualified  to  rate 
the  effectiveness  of  intent  statements. 

Taxonomy  of  functional  quality.  Through  interviews  and  review  of  relevant 
literature,  we  identified  several  elements  that  contribute  to  the  effectiveness  of  commander’s 
intent  statements.  We  asked  evaluators  to  rank  those  elements  in  terms  of  their  relative 
importance  to  them.  Using  the  median  to  measure  the  central  tendency  of  rank  for  each 
element  we  obtained  the  following  hierarchy: 

1.  Presents  a  clear  image  of  the  desired  outcome. 

2.  Provides  a  basis  for  determining  mission  priorities. 

3.  Presents  a  clear  image  of  how  the  brigade  mission  fits  into  the  division’s  effort. 

4.  Uses  clear  and  concise  language. 

5.  Does  not  present  conflicting  messages. 

6.  Does  not  provide  detailed  instructions  about  how  to  conduct  the  mission. 

We  also  asked  evaluators  to  identify  elements  of  an  intent  statement  that  should  be 
included  in  our  list.  They  reported  the  following: 

•  Intent  of  future  operations  if  operation  is  successful. 

•  How  the  commander  envisions  using  the  force  as  a  whole. 

•  Integrates  anticipated  enemy  disposition  and  action. 

•  Identifies  intent  for  each  subordinate  element/commander. 

•  Includes  use  of  deception  (tied  to  above). 

•  Presents  clear  statement  of  responsibilities  for  execution. 

•  Addresses  significant  risks  and  how  they  are  to  be  dealt  with. 

•  Identifies  and  describes  main  effort  and  conditions  under  which  it  would  be 
changed. 

•  Content  and  contributions  of  any  deception  efforts. 

•  Should  include  a  concise  statement  of  intent  of  division  and  corps  commanders, 
i.e.,  two  levels  up. 

General  Findings 

Several  important  findings  have  emerged  from  this  study.  First,  there  is  not  strong 
agreement  between  the  raters  on  the  coding  categories.  Although  there  is  sufficient 
agreement  to  permit  further  analyses,  it  is  noteworthy  that  the  raters  do  not  share  a  clear, 
common  understanding  of  what  is  involved  in  formulating  an  effective  commander  s  intent 
statement.  Furthermore,  the  raters  we  identified  as  the  most  experienced  agreed  with  each 
other.  Thus,  there  is  a  central  cluster  that  differs  from  the  others. 
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Second,  there  are  clear  differences  among  the  different  commanders.  Some  were 
seen  as  having  better  intent  statements  than  others.  More  interesting,  some  commanders 
show  improvement  from  the  beginning  to  the  final  exercises  at  NTC;  others  do  not  show 
much  improvement.  Moreover,  we  can  see  patterns  regarding  the  ways  that  units  do 
improve— how  their  intent  statements  change. 

Third,  the  aspect  of  commander’s  intent  that  is  identified  as  most  important  pertains 
to  the  visualization  of  the  batfiefield.  Some  of  the  NTC  units  failed  to  convey  an  adequate 
visualization,  while  others  were  successful. 

Fourth,  we  expected  that  excessively  detailed  intent  statements  would  be  rated  as 
poor,  since  the  detailed  account  of  how  to  accomplish  a  mission  is  inconsistent  with  the  need 
to  portray  a  visualization  of  the  outcome.  However,  the  intent  statements  rated  as  more 
detaUed  were  not  rated  as  less  effective.  The  only  problem  with  detailed  intent  statements 
was  that  they  appeared  to  engender  more  conflicts  and  internal  inconsistencies,  and  even  this 
may  have  been  useful  for  surfacing  areas  of  confusion  that  might  otherwise  have  been 
masked. 

We  have  identified  eight  functions  of  a  commander’s  intent  statement.  Such 
statements  should  enable  personnel  to: 

—  Understand  how  a  mission  objective  fits  in  with  the  plans  and  objectives  of  the 
higher— level  organization 

-  Prepare  to  continue  the  effort  after  the  objective  is  accomplished 

-  Establish  priorities,  to  know  how  to  invest  resources 

—  Determine  non-goals  at  points  of  ambiguity,  to  help  clarify  the  goals 

-  Direct  their  actions  without  needing  constant  guidance 

—  Improvise  as  necessary,  if  the  pre-conditions  change  drastically 

—  Recognize  when  pre-conditions  have  changed 

—  Recognize  unexpected  opportunities  and  judge  whether  to  pursue  them. 

We  have  reviewed  the  commander’s  intent  statement  as  a  frame,  containing  several 
slots.  Such  slots  might  include: 

-  the  mission  objective  (which  is  the  title  of  the  frame),  the  purpose  of  the  objective 

(e.g.,  the  higher-level  goal). 


37 


—  non-objectives  (to  clarify  ambiguous  aspects  of  the  objective), 

—  plan  features,  which  constitute  their  own  frame,  with  slots  for  the  actions,  the 
rationale  for  each  action,  contingencies,  and  additional  considerations  to  be  taken 
into  account  such  as  terrain 

__  criteria  and  priorities,  which  indicate  what  counts  as  a  successful  outcome  (e.g., 
enemy  casualties)  and  targeted  resources  for  achieving  the  outcome,  such  as  time  of 
expected  completion  and  casualty  rate  for  friendly  troops 

—  the  adversary’s  intent. 

We  have  found  that  virtually  all  of  the  intent  statements  describe  the  plan  of  action, 
and  29  of  the  35  intent  statements  include  a  rationale  for  plan  elements.  However,  only 
seven  statements  indicate  the  higher— level  purpose  for  the  plan.  Furthermore,  the  criteria 
for  achieving  the  plan  are  not  clearly  presented.  We  found  that  11  statements  indicated 
success  in  terms  of  enemy  casualties,  but  only  one  referred  to  guidance  on  friendly 
casualties.  None  of  the  statements  indicated  the  time  when  the  overall  objective  was  to  be 
accomplished.  In  most  organizations,  if  a  task  is  given,  there  is  the  expectation  that  one  will 
be  told  when  it  must  be  finished,  and  the  level  of  effort  to  expend.  However,  for  these 
statements  that  did  not  hold.  The  time  of  completion  and  level  of  resources  (e.g.,  personnel 
casualties)  were  missing.  It  is  possible  that  time  of  completion  was  stated  elsewhere  in  the 
operations  order,  but  there  seems  to  be  reluctance  to  estimate  acceptable  casualty  levels,  due 
to  the  effect  this  might  have  on  morale. 

These  findings  suggest  that  intent  statements  can  be  characterized  as  frames  with 
slots,  and  that  commanders  can  be  given  guidance  on  how  to  instruct  statements  that  are 
more  clear.  In  addition,  soldiers  can  be  taught  to  expect  certain  types  of  guidance  from  an 
intent  statement,  in  order  to  seek  clarification  if  necessary  information  is  missing. 

APPLICATIONS  OF  RPD  MODEL  AND  RESEARCH  ON  DECISION  MAKING 

In  this  section  we  review  a  series  of  projects  intended  to  apply  our  research  in 
decision  making. 

Studies  conducted  under  this  contract  have  extended  our  understanding  of  how 
decisions  are  made  in  operational  settings.  Our  findings  are  resulting  in  practical 
implications  for  the  development  of  training  programs  and  decision  support  systems. 

During  the  past  several  years,  we  have  been  testing  applications  of  the  RPD  model  and  the 
knowledge  elicitation  methods  we  developed  in  parallel  with  it.  The  applications  have  been 
carried  out  for  a  variety  of  government  and  commercial  sponsors  and  in  a  number  of 
different  tasks  and  domains. 
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One  primary  focus  of  our  appUed  work  is  designing  human-computer  interfaces 
(HCIs)  The  use  of  decision  strategies  to  identify  system  requirements  provides  a  different 
Uspective  for  designers,  to  enable  them  to  be  better  able  to  identify  cognitive  requirements. 
We  are  using  a  Decision-Centered  Design  approach  that  is  a  form  of  Cogmtive  Systems 
Engineering,  in  which  the  cognitive  task  analysis  centers  around  the  key  decision  points,  and 
attempts  to  describe  the  types  of  strategies  used  to  make  these  decisions.  This  approach  has 
been  used  in  a  project  for  the  Armstrong  Laboratory,  in  designing  the  interface  for  the 
Self-Defense  Suite  of  the  JSTARS  aircraft  (Thordsen,  Wolf,  &  Crandall,  1990),  a  joint 
Army-Air  Force  surveillance  aircraft.  We  used  the  RPD  understand 

strategies.  Another  appUcation  is  for  the  Navy,  concerning  the  HCI  of  the  AEGIS  Combat 
Information  Center  to  develop  a  set  of  design  principles  for  use  m  development  decision 
support  systems.  Specifically,  the  project  on  Tactical  Decision  Making  Under  Stress 
(TADMUS)  is  aimed  at  understanding  how  crew  members  in  a  CIC  are  able  to  make 
complex  decisions  when  faced  with  time  pressure  and  other  stressors.  TADMUS  appUes  the 
RPD  model  and  similar  approaches  to  naturalistic  decision  making. 

Another  application  of  our  decision  research  is  to  understand  consumer  decision 
making.  We  have  conducted  in-depth  market  research  studies  for  Johnson  and  Johnson,  and 

for  Procter  and  Gamble. 

A  third  area  of  application  is  for  team  training.  We  have  conducted  a  number  of 
studies  to  develop  methods  for  observing  and  providing  feedback  to  teams.  These  have 
included  echelons  above  corps,  at  the  Army  War  College,  the  Industrial  College  of  the 
Armed  Forces,  and  the  Air  Force  Institute  of  Technology  (Klein,  Zsambok,  Kyne,  & 
Klinger,  1991),  battalions  (Thordsen,  Klein,  &  Wolf,  1991),  and  helicopter  crews  (Thordsen 
&  Klein,  1991).  Other  applications  have  been  for  emergency  operations  teams  in 
manufacturing  facilities  (Klein,  Thordsen,  &  Calderwood,  1989)  and  a  current  effort 
sponsored  by  the  FAA  to  apply  our  decision  research  to  Advanced  Qualification  Program 

(AQP). 


SUMMARY 

There  are  a  number  of  major  accomplishments  emerging  from  this  Basic  Res^ch 
contract.  These  have  been  identified  in  the  earlier  sections  of  this  Final  Report,  and  we  will 
simply  review  them  here. 

First,  the  research  performed  and  projects  accomplished  have  establish^  Naturalistic 
Decision  Making  as  an  important  and  unique  perspective.  The  previous  traditions  of 
classical  decision  research,  drawing  on  mathematics,  statistics,  and  game  theory,  have 
definite  strengths  and  wealoiesses,  and  there  is  now  emerging  an  acknowledgement  of  the 
importance  of  Naturalistic  Decision  Making  for  addressing  the  problems  and  requirements 
that  are  not  handled  well  by  the  classical  research  strategies.  The  1989  workshop  on  ^M 
was  extremely  significant  for  solidifying  this  approach,  and  the  forthcoming  book  Decisioii 
Makinp  in  Action-  Models  and  Methods  will  continue  that  process.  Without  these  projects. 
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the  earlier  research  findings  would  likely  have  remained  on  the  periphery  of  the  decision 
research  community.  Because  of  these  projects,  the  direction  of  decision  research  may  have 
been  permanently  changed.  We  believe  that  there  has  been  a  breakthrough  m  cognitive 
accounts  of  decision  making,  and  because  of  this  research  proj^t  we  have  been  able  to 
solidify  that  breakthrough  and  make  it  accessible  to  other  cognitive  researchers.  We  also 
believe  that  this  breakthrough  wiU  have  important  applications  for  decision-centered  design  of 
interfaces  and  displays,  training  that  emphasizes  situation  assessment,  mental  simulation  and 
metacognition  rather  than  analytical  decision  strategies,  and  more  effective  approaches  to 
team  decision  training. 

Second,  we  have  demonstrated  that  the  RPD  model  generates  empirical  as  well  as 
ethnographic  research.  While  the  RPD  model  was  developed  through  ethnographic  research 
methods,  currently  there  have  been  as  many  empirical  studies  of  the  model  as  ethnographic 
studies  The  field  studies  have  taken  place  in  diverse  settings  (urban  firefighting,  forest  fire 
management,  design  engineering,  tank  platoon  operations,  nursing  in  Intensive  Care  Units), 
and  the  incident  accounts  from  these  studies  have  been  vivid  and  memorable,  so  it  may  be 
easy  to  forget  the  empirical  studies.  Thus  far  there  have  been  at  least  a  half-dozen  studies  of 
the  RPD  model.  We  have  examined  firefighters,  using  multi-dimensional  scaling,  we  have 
used  prepared  scenarios  to  collect  decision  protocols  from  experienced  and  inexperienced 
fireground  commanders.  Hosier  (1990)  has  confirmed  the  use  of  recogniUonal  decision 
making  in  airline  pilots,  using  analyses  of  videotapes  collected  during  high  fidelity 
simulations.  We  have  shown  that  proficient  performance  can  be  maintained  by  skilled  chess 
players,  despite  severe  time  constraints.  In  the  next  few  months  we  also  expect  to  show  that 
skilled  chess  players  generate  superior  options  as  the  first  ones  they  consider,  a  direct  test  of 
the  RPD  model.  We  have  also  shown  that  mental  simulation  exercises  can  improve  decision 
processes  of  college  students  engaged  in  planning  tasks. 

Third,  through  the  mix  of  ethnographic  and  empirical  research,  the  Recognition- 
Primed  Decision  model  has  become  more  mature  and  comprehensive  than  it  was  three  years 
ago,  at  the  beginning  of  this  contract.  It  has  become  elaborated  to  portray  several  means  of 
achieving  situation  assessment.  It  has  been  shown  to  generate  testable  hypotheses,  and  it  has 
been  experimentally  supported. 

Fourth,  the  primary  expansion  of  the  RPD  model  has  been  to  achieve  a  greater 
understanding  of  the  process  of  mental  simulation.  The  current  research  effort  has 
accomplished  that  important  objective.  We  have  derived  models  that  are  far  more  detailed 
than  the  ones  previously  available,  and  we  have  shown  how  mental  simulation  serves  as  a 
source  of  power  for  a  variety  of  cognitive  functions. 

The  naturalistic  decision  making  (NDM)  perspective  has  shown  a  great  deal  of  - 
progress  in  the  short  time  that  it  has  guided  field  researchers.  As  expected,  linkages  with 
other  cognitive  processes  such  as  planning  and  decision  making  have  been  facilitat^  by  the 
shift  from  a  classical  decision  perspective.  Models  have  begun  to  proliferate,  but  in  contact 
with  others  to  permit  synthesis  as  we  learn  more  about  boundary  conditions.  The  RPD 
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model  itself  has  undergone  a  steady  elaboration,  maintaimng  its  overall  structure  while 
increasing  the  detail  with  which  it  describes  sub-processes  such  as  situation  assessment  and 
mental  simulation.  Each  new  perspective  brings  with  it  a  period  of  rapid  expansion  of 
knowledge,  followed  by  a  slowing  of  accomplishments  as  the  field  reaches  an  asymptote  and 
the  questons  addressed  become  less  productive.  It  appears  as  if  the  NDM  perspective  is  still 
within  the  initial  phase  of  rapid  exploration  and  appUcation. 

REFERENCES 

Beach,  L.  R.,  &  Lipshitz,  R.  (in  press).  A  critique  of  classical  decision  thwry  as  a 
behavioral  theory.  In  G.  A.  Klein,  J.  Orasanu,  R.  Calderwood,  and  C  E.  Z^bok 
(Eds.),  Dp-eisinn  making  in  action:  Models  and  methodj.  Norwood,  NJ.  Ablex 

Publishing  Corporation. 

Beach,  L.  R.,  Mitchell,  T.  R.  (1978).  A  contingency  model  for  the  selection  of  decision 
strategies.  Academy  of  Management  Review,  3,  439-449. 

Black,  M.  (1962).  MnHe.ls  and  metaphors.  Ithaca,  NY:  ComellUniversity  Press. 

Calderwood,  R.  (1989).  The  role  of  context  in  modeling  domain  knowledge.  Unpublished 
doctoral  dissertation,  University  of  New  Mexico,  Albuquerque,  NM. 

Calderwood,  R.,  Crandall,  B.,  &  Klein,  G.  A.  (1987).  Expert  and  novice  firy round 

rnmmand  decisions  (KATR-858(D)-87-02F).  Yellow  Sprmgs,  OH:  IQein  Associates  Inc. 
Prepared  under  contract  MDA903-85-C-0327  for  U.  S.  Army  Research  Institute, 
Alexandria,  VA.  AD  A225666 

Calderwood,  R.,  Klein,  G.  A.,  &  Crandall,  B.  W.  (1988).  Time  pressure,  skill,  and  move 
quality  in  chess.  American  Journal  of  Psychology.  481-493. 

Cohen,  M.  S.  (in  press).  The  naturalistic  basis  of  decision  biases.  In  G.  A.  Klein, 

J.  Orasanu,  R.  Calderwood,  &  C.  E.  Zsambok  (Eds.),  Uecision  making  m  action:  Models 
and  methods.  Norwood,  NJ:  Ablex  Publishing  Corporation. 

de  Groot,  A.  D.  (1965/1978).  Thought  and  choice  in  chess  (2nd  ed.).  NY:  Mouton. 

DriskeU,  J.  E.,  &  Salas,  E.  (in  press).  Can  you  study  real  teams  in  contrived  settings?  The 
value  of  small  group  research  to  understanding  teams.  In  R.  Swezey  and  E.  Salas  (Eds.), 
Teams:  Their  Training  and  Performance.  Norwood,  NJ :  Ablex  Publishing  Corporation. 

DriskeU,  J.  E.,  &  Salas,  E.  (1991).  Overcoming  the  effects  of  stress  on 

performance:  Human  factors  design,  training,  and  selection  strategies.  In  R*  Gal  ^d 
A.  D.  Mangelsdorff  (Eds.),  Handbook  of  Military  Psychology.  NY:  John  Wiley  & 

Sons. 


41 


Duncker,  K.  (1945).  On  problem  solving.  Psychological  Monographs,  58(5,  Whole  No. 
270). 

Einhom,  H.  J.,  &  Hogarth,  R.  M.  (1981).  Behavioral  daision  tteory:  Processes  of 
judgment  and  choice.  Annual  Review  of  Psychology,  32,  53-88. 

Einhom,  H.  J.,  &  Hogarth,  R.  M.  (1987).  Adaptotion  and  inertia  in  brief  updaring  (Tech. 
Rep.).  University  of  Chicago;  Center  for  Decision  Research. 

Fleiss,  J.  L.  (1981).  Statistical  methods  for  rates  and  proportion^.  (2nd  ed.)  NY:  John 
WUey. 

Gettvs  C.  F.  (1983).  Ppsenrch  and  theory  on  predecision  processes  (TR  11-30-83). 
Norman,  OK:  University  of  Oklahoma,  Decision  Processes  Laboratory. 

Hammond,  K.  R.,  &  Adelman,  L.  (1976).  Science,  values,  and  human  judgement.  Scmce. 
194,  389-396. 

Hesse,  M.  (1966).  MndeJs  and  analogies  in  science.  Notre  Dame:  University  of  Notre 
Dame  Press. 


Hogarth,  R.  M.  (1986).  Generalization  in  decision  research:  The  role  of  formal  models. 
TF.F.F.  Sv.stems.  Man,  and  Cybernetics,  SMC- 16,  439-449. 

Holding,  D.  H.  (1985).  The  osvcholopv  of  chess  skill.  Hillsdale,  NJ;  Erlbaum. 

Jungermann,H.,&Thuring,M.  (1987).  The  use  of  causal  knowledge  for  inferential 

reasoning,  NATO  ASI  Series,  £35,  13 1- 146. 

Kaempf  G.  L.,  Klein,  G.,  &  Kyne,  M.  M.  Hn  preparation).  Effectiveness  of  Commander’s 
Intent  .Statements  in  Onerations  Orders.  Fairborn,  OH;  Klein  Associates  Inc. 

TCahneman  D  &  Tverskv  A.  (1982).  On  the  study  of  statistical  intuitions.  In 

D.  Kahnem’an,  P.  Slovic,  &  A.  Tversky  (Eds.),  Judgment  under  uncertainty  -  heunstics 
and  biases.  Cambridge:  Cambridge  University  Press. 

Klein,  G.  (1992).  Frrnr«;  in  naturalistic  decision  making.  Fairborn,  OH:  Klein  Associates 
Inc. 


Klein,  G.,  &  Miller,  T.  E.  (1992).  A  convergence  model  of  option  generation.  Fairborn 
OH;  Klein  Associates  Inc. 


42 


Klein  G.,  Wolf,  S.,  Militello,  L.,  &  Zsambok,  C.  (1992).  Characteristics  of  skilled  option 
generation  in  chess.  Submitted  for  publication  to  the  Journal  of  Experimental 
Psychology. 

Klein  G  &  Zsambok,  C.  E.  (in  preparation).  The  effect  of  acute  stressors  on  decision 
mking.  In  J.  E.  Driskell  and  E.  Salas  (Eds.),  Stress  and  human  performan^.  Hillsdale, 
NX:  Lawrence  Erlbaum  Associates,  Inc.  Fairborn,  OH;  Klein  Associates  Inc. 


Klein  G.  A.  (1989).  Recognition-primed  decisions.  In  W.B.  Rouse  (Ed.),  Advancesjn 
Man-Machine  System  Research.  5,  47-92.  Greenwich,  CT;  JAI  Press,  Inc. 


Klein,  G.  A.  (1990a).  Mental  simulation  and  decision  making.  Proceedings  of  the  Fifth 
Mid-Central  Erponomics/Human  Factors  Conference.  Dayton,  OH. 


Klein,  G.  A.  (1990b).  Knowledge  engineering;  Beyond  expert  systems.  In  M.  Singh  and 
A.’  Sage  (Eds.),  Information  and  decision  technologies,  16(1).  Holland;  Elseyier 

Science. 

Klein  G.  A.  (1991).  Effects  of  stress  on  decision  making.  Proceedings  of  3rd  TopicM 
Meeting  on  Emergency  Preparedness  and  Response,  American  Nuclear  Society,  Apn  . 


Klein  G.  A.,  Calderwood,  R.,  &  Clinton-Cirocco,  A.  (1985).  Rapid  decision  matang  on 
the  fire  ground  (KA-TR-84-41-7).  Yellow  Springs,  OH;  Klein  Associates  Inc.  Prepared 
under  contract  MDA903-85-G-0099  for  the  U.S.  Army  Research  Institute,  Alexandna, 
VA.  A.D  A 19949 


Klein,  G.  A.,  Calderwood,  R.,  &  Clinton-Cirocco,  A.  (1986).  Rapid  decision  m^ng  o 
the  fire  ground.  Proceedings  of  the,  3nth  Annual  Human  Factors  Society,  1,  576-580. 
Dayton,  OH;  Human  Factors  Society. 

Klein  G  A  &  Crandall,  B.  (in  preparation).  Mental  simulation.  In  J.  Flach,  P. 

H^cock,  J.  Caird,  and  K.  Vicente  (Eds.),  The  ecology  of  human-machine  systems. 

Klein  G.  A.,  Orasanu,  J.,  Calderwood,  R.,  &  Zsambok,  C.  E.  (in  press)  D^ision 

making  in  action-  Models  and  methods.  Norwood,  NJ;  Ablex  Publishing  Corporation. 

Klein  G.  A.,  &Thordsen,  M.  L.  (1989).  Cognitiye  processes  of  the  team  mjnd.  Final 
R^ort  submitted  to  NASA-Ames  Research  Center  under  PO#A72145C. 

Klein,  G.  A.,  Thordsen,  M.,  &  Calderwood,  R.  (1989).  Eyaluation  of  cnsis  deasion 
making  during  training  scenario.  May  18,  1989,  Security  Crisis  M^agement  H  course  a 
the  Central  Training  Academy.  Working  paper.  Yellow  Spnngs,  OH;  Klein  Associates 

Inc. 


43 


Klein,  G.  A.,  &  Weitzenfeld,  J.  (1978).  Improvement  of  skills  for  solving  ill-defined 
problems.  Educational  Psychologist.  13,  31-41. 

Klein  G.  A.  &  Zsambok,  C.  E.  (1991).  Models  of  skilled  decision  making.  Proceedings 
of  the  1991  Human  Factors  Society  35th  Annual  Meeting,  1363-1366.  Santa  Momca, 

CA:  Human  Factors  Society. 

Kvne  M.,  Militello,  L.,  &  Klein,  G.  A.  (in  preparation).  The  effects  of  mental  simulation 
nn  confidence.  Working  paper.  Fairborn,  OH:  Klein  Associates  Inc. 

Means,  B.,  Salas,  E.,  Crandall,  B.,  &  Jacobs,  O.  (in  press).  Trmning  d^ision  in^aters  for 
the  real  world.  In  G.  A.  Klein,  J.  Orasanu,  R.  Calderwood,  &  C.  E.  Zsambok  (^s.), 
making  in  action:  Models  and  method_s.  Norwood,  NJ:  Ablex  PubUshing 

Corporation. 

Montgomery,  H.  (in  press).  The  search  for  a  dominance  structure  in  drasion  maMng: 
Examining  the  evidence.  In  G.  A.  Klein.  J.  Orasanu,  R.  Calderwood  and  C  E. 
Zsambok  (Eds.),  Decision  making  in  action:  Models  and  methods.  Norwood,  NJ. 

Ablex  Publishing  Corporation. 

Mosier,  K.  L.  (July,  1990).  Decision  making  on  the  air  transport  flight  deck:  Process  and 
product.  Unpublished  dissertation.  University  of  California,  Berkeley,  CA. 

Orasanu,  J.,  &  Connolly,  T.  (in  press).  The  reinvention  of  decision  inaking.  In  G.  Klein, 
J.  Orasanu,  R.  Calderwood,  &  C.  E.  Zsambok  (Eds.),  Decision  making  in  action^ 
Models  and  methods.  Norwood,  NJ;  Ablex  Publishing. 

Orasanu  J.,  &  Salas,  E.  (in  press).  Team  decision  making  in  complex  enviromnent.  In 
G  IQein,  J.  Orasanu,  R.  Calderwood,  &.  C.  E.  Zsambok  (Eds.),  Decision  making  m 
acrinn:  Models  and  methods.  Norwood,  NJ:  Ablex  Publishing. 

Payne  J.  W.  (1976).  Task  complexity  and  contingent  processing  in  decision  making.  An 
information  search  and  protocol  analysis.  Organizational  Behavior  and  Human 
Performance.  16.  366-387. 

Payne  J  W.,  Bettman,  J.  R.,  &  Johnson,  E.  J.  (1988).  Adaptive  strategy  selection  in 
decisionmaking.  Toumal  of  Experimental  Psychology:  Learning,  Memory,  and 
Cognition.  14(3).  534-552. 

Pennington,  N.,  &  Hastie,  R.  (in  press).  A  theory  of  explanation-based  decision  making. 
In  G  Klein,  J.  Orasanu,  R.  Calderwood,  &  C.  E.  Zsambok  (Eds.),  Decision  making  in 
action:  Models  and  methods.  Norwood,  NJ:  Ablex  Publishing. 


44 


Rasmussen,  J.  (1985).  The  role  of  hierarchical  knowledge  representation  in  decision  making 
and  system  management.  THEE  Transactions  on  Systems,  Man  and  Cybemetics,  SMCz 
15(2),  234-243. 


Robinson,  J.  A.,  &  Hawpe,  L.  (1986).  Narrative  thinking  as  a  heuristic  process.  In 
T.  Sarbin  (Ed.),  Narrative  psychology:  The  storied  nature  of  human  conduct.  NY. 

Praeger  Publishers. 


Rouse,  W.  B.,  &  Morris,  N.  M.  (1986). 
limits  on  the  search  for  mental  models. 


On  looking  into  the  black  box;  Prospects  and 
Psychological  Bulletin.  100(3),  349-363. 


Smith,  P.  J.,  Galdes,  D.,  Fraser,  J.,  Miller,  T.,  Smith,  J.  W.,  Jr.,  Svirbely,  J.  R.,  Bl^ina, 
J.  Kennedy,  M.,  Ruddman,  S.,  &  Thomas,  D.  L.  (1991).  Coping  with  the  complexities 
of  multiple-solution  problems:  A  case  study.  In  International  Journal  Man-Machme 
Studies.  35.  429-453. 


Soelberg,  P.  O.  (1967).  Unprogrammed  decision  making.  Industrial  Management  Review, 
8,  19-29. 

Svenson,  O.  (1979).  Process  descriptions  in  decision  making.  Organizational  Behavior  and 
Human  Performance.  23,  86-112. 


Thordsen,  M.  L.  (1992).  A  test  of  a  team  performance  error  taxonomy:  Including 
prelim’inary  examination  of  the  relationship  between  domain  attribute  and  error 
distribution.  Submitted  for  publication  to  IEEE. 


Thordsen,  M.,  Galushka,  J.,  Klein,  G.  A.,  Young,  S.,  &  Brezovic,  C- P-  (  )•  — 

Vnnwiedge  elicitation  study  of  military  planning.  Technical  Report  867.  Alexandna,  VA. 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. AD  219  969 


Thordsen,  M.L.,  &  Klein,  G.  A.  (1989).  Cognitive  processes  of  the  team  mind. 
Prnrpftiinps  of  IEEE  International  Conference  on  Systems,  Man,  and  Cybemetics. 
Cambridge,  MA. 

Thordsen,  M.  L.,  &  Klein,  G.  A.  (1991).  Training  implications  of  a  team  decision  model. 
Proceedings  of  the  Sixth  Industrial  Symposium  on  Aviation  Psychology,  Columbus,  OH. 

Thordsen,  M.  L.,  Klein,  G.  A.,  &  Wolf,  S.  P.  (1991).  Using  a  cognitive  model  to  evaluate 
tp^^mwnrk  (Final  Technical  Report).  Fairborn,  OH:  Klein  Associates  Inc.  Prepared 
under  contract  MDA903-90-C-0119  for  the  U.S.  Army  Research  Institute  for  the  • 
Behavioral  and  Social  Sciences,  Alexandria,  VA. 


45 


Thordsen,  M.  L.,  Wolf,  S.  P.,  &  Crandall,  B.  (1990).  TTser-defin^  r^uirements^^^^^^^^ 
t5;tair.S  self-Hp.fp-nse  suite.  Final  report  prepared  under  Contract  F33615-88-D  0536  m 
coniunction  with  the  UniTys  Corporation  for  the  Armstrong  Aerospace  Medical  Research 
Laboratory,  Human  Systems  Division,  Air  Force  Systems  Command,  Wnght-Patterson 
APB,  OH.  Dayton,  OH:  Unisys  Corporation. 

Tikhomirov,  O.  K.,  &  Poznyanskaya,  E.  D.  (1966).  An  “ 

means  of  analyzing  heuristics.  Soviet  Psychology.  5(3-15),  53,  54,  55,  57,  126,  18  , 

185,  187). 

Zsambok,  C.  E.,  Klein,  G.,  Kyne,  M.  M.,  &  Klinger,  D.  W.  £992)  Advanc^^ 
dpri«;ion  making:  A  developmental  model.  Fairborn,  OH:  Klein  Associat^  Inc. 

Prepared  by  Klein  Associates  for  the  U.S.  Army  Research  Institute  for  the  Behavioral  and 

Social  Sciences,  Contract  MDA903-90-C-0 117.  ad  A289  855 


46 


appendix  a 


"Recognition-Primed  Decision  Making" 
Contract  No.  MDA903-89-C-(X)32 
Supporting  Documentation 


Calderwood,  R.  (1988,  August).  Fireground  decision  making;  A  view  from  the  outside. 

Fire  Command,  pp.  34-40. 

Calderwood,  R.  (1988).  The  role  of  context  in  modeling  domain  knowledge.  Unpublished 
doctoral  dissertation.  University  of  New  Mexico,  Albuquerque,  NM. 

Kaempf,  G.  L.  (September,  1991).  Expert  decision  making  under  stress.  Proceedings  of 

the  Rth  Users’  Stress  Workshop.  San  Antonio,  TX. 

Kaempf,  G.  L.,  &  Kyne,  M.  M.  (in  preparation).  Commander’s  intent.  Fairborn,  OH: 
Klein  Associates  Inc. 

Klein,  G.  A.  (1991).  Invited  speaker,  AFIT  Commencement  Exercise,  March. 

Klein  G.  A.  (in  press).  A  recognition-primed  decision  (RPD)  model  of  rapid  decision 
midng.  In  G.  A.  Klein,  J.  Orasanu,  R.  Calderwood,  and  C.  E.  Zsambok  (Eds.), 
r>pri<;inn  making  in  action:  Models  and  methods.  Norwood,  NJ:  Ablex  Publishing 

Company. 

Klein,  G.  A.  (in  press).  Twenty  questions;  Suggestions  for  research  in  naturalistic  decision 
mking.  In  G.  A.  Klein,  J.  Orasanu,  R.  Calderwood,  and  C.  E.  Zsambok  (Ms.), 
Derisinn  makipp  in  action:  Models  and  methods.  Norwood,  NJ:  Ablex  Publishing 

Company. 

Klein  G  A.  (1991).  Effects  of  stress  on  decision  making.  Accepted  for  publication  in 
Prnrpedings  of  3rH  Tnnical  Meeting  on  Emergency  Preparedness  and  Response,  American 

Nuclear  Society,  April. 

Klein,  G.  A.  (May,  1990).  Mental  simulation  and  decision  making.  Proceedings  of  the 
Fifth  Mid-Central  Ergonomics/Human  Factors  Conference.  Dayton,  OH. 

Klein,  G.  A.,  &  Calderwood,  R.  (in  press).  Decision  Models;  Some  lessons  from  the 
field.  In  A.  Sage  (Ed.),  IEEE  Transactions  on  Systems,  Man,  and  Cybernetics. 

Klein,  G.  A.,  Crandall,  B.  W.,  &  Kyne,  M.  M.  (in  preparation).  Evaluation  of  failure 
analysis  on  confidence.  Fairborn,  OH;  Klein  Associates  Inc. 


47 


Klein  G.  A.,  &  Hoffman,  R.  (in  press).  Seeing  the  invisible:  PerceptuM^ognitive  aspects 
of  expertise.  In  M.  Rabinowitz  (Ed.),  Applied  Cognition.  Norwood,  NJ:  Ablex 
Corporation. 

Klein,  G.  A.,  &  Klinger,  D.  W.  (1991).  Naturalistic  decision  making.  CSERIAC, 

Gateway.  2(1),  1-4. 

Klein  G.  A.,  Orasanu,  J.,  Calderwood,  R.,  &  Zsambok,  C.  E.  (in  press)  D^ision 
m^vinp  in  action:  Models  and  methods.  Norwood,  NJ:  Ablex  PubUshmg  Corporation. 

Klein,  G.  A.,  &  Thordsen.  M.  L.  (1991).  Representing  cockpit  otw  decision  making. 

Ciyth  Tntemarional  .Symposium  on  Aviation  PsvchoIogY.  April  29-May  2,  1991, 

Columbus,  OH. 

Klein,  G.  A.,  &  Thordsen,  M.  L.  (1991).  Training  impUcations  of  ^  decisbn  models. 

Sixth  International  Symposium  on  Aviation  Psychology,  Apnl  29-May  2,  1991, 

Columbus,  OH. 

Klein,  G.  A.,  Woods,  D.  D.,  &  Orasanu,  I.  (in  press).  Conclusions  In  G.  A.  Klein  J 
Orasanu,  R.  Calderwood,  and  C.  E.  Zsambok  (Eds.),  Decision  makinp  m  action.  Models 
and  methods.  Norwood,  NJ:  Ablex  Publishing  Corporation. 

Klein,  G.  A.,  &  Zsambok,  C.  E.  (1991).  Models  of  skiUed  decision  making.  Proceedings 
of  the  Human  Factors  Society  35th  Annual  Meeting,  1363-1366. 

Means,  B.,  Salas,  E.,  Crandall,  B.,  &  Jacobs,  O.  (in  press)  Training  dmsion  ^aken  for 
the  real  world.  In  G.  A.  Klein,  J.  Orasanu,  R.  Calderwood,  and  C.  E.  Z^bok  (Eds.), 
Deoi^iinn  makinv  in  action:  Models  and  methods.  Norwood,  NJ:  Ablex  Publishing 

Company. 

Rothman,  J.,  &  Klein,  G.  A.  (in  preparation).  Why  conventional  negotiation  methods 
eannot  handle  ‘intractable’  conflicts.  Submitted  for  publication  to  Negotiation  JoumM. 

Thordsen,  M.L.,&  Klein,  G.  A.  (1989).  Cognitive  processes  of  the  te^  mind 

Proceedings  nf  TEER  International  Conference  on  Systems,  Man,  and  Cybernetics,  1,  46  49. 

Cambridge,  MA. 

Wolf,  S.  P,  (In  preparation).  An  examination  of  decision  option  generation  strategies. 
Fairborn,  OH:  Klein  Associates  Inc. 

Zsambok,  C.  E.,  Klein,  G.,  Kyne,  M.  M.,  &  Klinger,  D.  W.  0992)  Advanced  team 
decision  making:  A  developmental  model.  Fairborn,  OH:  Klein  Associat^  Itac. 

Prepared  by  Klein  Associates  for  the  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  Contract  MDA903-90-C-0117.  AD  A289855 


48 


